There’s a problem with use of generative AI chatbots that I think isn’t getting enough attention.
Recently I saw someone had posted an issue against an open source project, saying that the maintainer’s use of generative AI to create code could lead to intellectual property issues. The maintainer did what you might expect — they asked their favorite cutting-edge AI chatbot for a summary of the situation.
I’m not going to name the project. I don’t want to be responsible for brigading, because this is happening everywhere. The identity of the project and maintainer also aren’t relevant to the point I’m about to make. It’s not even about software, particularly, that’s just the example I happened to see.
Anyway, the bot did what LLM bots do — it produced random text that looked statistically like a response to the question asked. It did this based on statistical analysis of text of the most common responses to similar questions found in its training set. The result looked very convincing. It said that work partially generated by AI can be protected by copyright if there is sufficient human input.
Unfortunately, it missed a critical detail. The US copyright office guidance is that you must state which parts of the work were AI generated, and explicitly exclude those from any copyright claim.
Quoting Copyright Registration Guidance: Works Containing Material Generated by Artificial Intelligence:
Individuals who use AI technology in creating a work may claim copyright protection for their own contributions to that work. They must use the Standard Application, and in it identify the author(s) and provide a brief statement in the “Author Created” field that describes the authorship that was contributed by a human. […] AI-generated content that is more than de minimis should be explicitly excluded from the [copyright] application.
I’ve not seen any projects actually do this. They all seem to be trying to slap a blanket copyright license over everything, even AI-generated material that can’t be copyrighted. This is not a new problem, and not specific to software — companies have issued DVDs of out-of-copyright movies and tried to claim copyright on them; same with books.
Anyway, going back to the bot’s generated output, there’s no real mystery why this omission occurred. AI proponents are generally either unaware of the US Copyright Office position, or choose not to mention it because it’s inconvenient to them. Hence the bot’s training data either didn’t mention it anywhere, or mentioned it very rarely. Hence the bot’s output, which is based on picking the most probabilistically likely output for the given input, didn’t mention it.
Back when I was at school we were taught how to summarize a piece of text. The first step is to read through the text carefully, and note which details must be included in the summary. This typically requires some domain knowledge, or at least everyday experience of the real world. In business, you need to know the target audience, and understand what they will need (and want) to know. These are not generally things that can be derived using pure probabilistic analysis; you need to understand the text, and need a mental model of the audience. I’ve seen it said that LLMs don’t actually summarize text, they make it shorter, and those are different things.
I do actually use the “summarize” feature of Apple Intelligence. When I bookmark something, I’ll take the text of the page and “summarize” into a short paragraph or two. This is good enough to help me find the bookmark again when I search, but I’m under no illusion that it actually represents the content accurately. (Pre-LLM summarization algorithms were no better in this respect.)
I also use local neural network models for image recognition, to produce short descriptions for my photos to help me find them again. Again, the output can be pretty inaccurate, but it doesn’t matter too much; it just needs to include enough of the words I’m likely to use when searching.
I think we’re getting to the point where most people are aware that AI chatbots sometimes produce text that looks plausible, but is not in agreement with reality. Those who enthusiastically use AI will tell you that it’s OK, because they check the output. I don’t believe them, but let’s imagine they do. The problem is this: you can check the output for errors, but you can’t check for things that are missing unless you basically already know what the output should be. And if you know that, what’s the point of asking the AI for the answer?
As Donald Rumsfeld might have put it, the problem is the unknown unknowns. It doesn’t help if the bot provides exact citations for where it found the information, and it doesn’t help if you diligently check those citations. If the bot omits something entirely, you won’t know. It’s like the joke “Everyone who couldn’t make it to the meeting, please raise your hand”.
This issue of missing data in AI output is a problem for summaries of contracts and other legal documents. It’s a problem for AI analysis of email, documentation, scientific data, basically anything where you’re asking a bot for a non-trivial answer to a question you don’t already know the answer to.
It also doesn’t seem to me to be something that can be fixed using LLM technology. There are always going to be places where the training data doesn’t repeatedly emphasize something that a human would pick up on as being very important. This is particularly true when the subject matter has undergone a lot of changes in recent years.
So if you need an accurate answer, you’re going to have to ask a knowledgeable human. Or, you know, actually read some source material. Just checking the accuracy of a chatbot’s output doesn’t avoid the problem of incorrect answers.