Wikipedia talk:Large language models

This section is pinned and will not be automatically archived.

I wrote a user script called WikiChatbot. It works by selecting text in an article and then clicking one of the buttons on the right to enquire about the selected text. It includes many functions. For example, it can summarize and copyedit the selected text, explain it, and provide examples. The chat panel can also be used to ask specific questions about the selected text or the topic in general. The script uses the AI model GPT 3.5. It requires an API key from OpenAI. New OpenAI accounts can use it freely for the first 3 months with certain limitations. For a more detailed description of all these issues and examples of how the script can be used, see the documentation at User:Phlsph7/WikiChatbot.

I was hoping to get some feedback on the script in general and how it may be improved. I tried to follow WP:LLM in writing the documentation of the chatbot. It would be helpful if someone could take a look to ensure that it is understandable and that the limitations and dangers are properly presented. I also added some examples of how to use edit summaries to declare LLM usage. These suggestions should be checked. Feel free to edit the documentation page directly for any minor issues. I'm also not sure how difficult it is to follow the instructions so it would be great if someone could try to set up the script, use it, and explain which steps were confusing. My OpenAI account is already older than 3 months so I was not able to verify the claims about the free period and how severe the limitations are. If someone has a younger account or is willing to open a new account to try it, that would be helpful.

Other feedback on the idea in general, on its problems, or on new features to implement is also welcome. Phlsph7 (talk) 12:45, 12 July 2023 (UTC)Reply

I meant to reply to this sooner. This is awesome and I'm interested in this (and related ideas) related to writing / reading with ML. I'll try to have a play and give you some feedback soon. Talpedia 10:18, 17 July 2023 (UTC)Reply

Related: see also m:ChatGPT plugin. Mathglot (talk) 07:22, 18 July 2023 (UTC)Reply

Whilst I rather like the ability of this nifty little script to do certain things, I do have some criticism. These functions strike me as extremely risky, to the point that they should probably be disabled:

• "is it true?" - ChatGPT likely uses Wikipedia as a source, and in any case, we want verifiability, not truth. I feel quite strongly, based on several other reasons too, that this function should be disabled and never see the light of day again.
• "is it biased?" - ChatGPT lacks the ability to truly identify anything more than glaring "the brutal savages attacked the defenceless colonist family" level bias (i.e. something that any reasonably aware human should spot very quickly indeed). Best left to humans.
• "is this source reliable?" - Same as the first one, this has so much potential to go wrong that it just shouldn't exist. Sure it might tell you that Breitbart or a self-published source isn't reliable, but it may also suggest that a bad source is reliable, or at least not unreliable.

I don't think that any amount of warnings would prevent misuse or abuse of these functions, since there will always be irresponsible and incompetent people who ignore all the warnings and carry on anyway. By not giving them access to these functions, it will limit the damage that these people would cause. Doing so should not be a loss to someone who is using the tool responsibly, as the output generated by these functions would have to be checked so completely that you might as well just do it without asking the bot.

The doc page also needs a big, obvious warning bar at the top, before anything else, making it clear that use of the tool should be with considerable caution.

The doc page also doesn't comment much on the specific suitability of the bot for various tasks, as it is much more likely to stuff up when using certain functions. It should mention this, and also how it may produce incorrect responses for the different tasks. It also doesn't mention that ChatGPT doesn't give wikified responses, so wikilinks and any other formatting (bolt, italics, etc) must be added manually. The "Write new article outline" function also seems to suggest unencyclopaedic styles, with a formal "conclusion", which Wikipedia articles do not have.

Also, you will need to address the issue of WP:ENGVAR, as ChatGPT uses American English, even if the input is in a different variety of English. Mako001 (C)  (T)  🇺🇦 01:14, 23 July 2023 (UTC)Reply

You can ask it return wikified responses and it will do it with reasonable good success rate. -- Zache (talk) 03:03, 23 July 2023 (UTC)Reply

@Mako001 and Zache: Thanks for all the helpful ideas. I removed the buttons. I gave a short explanation at Wikipedia:Village_pump_(miscellaneous)#Feedback_on_user_script_chatbot and I'll focus here on the issues with the documentation. I implemented the warning banner and add a paragraph on the limitations of the different functions. That's a good point about the English variant being American so I mentioned that as well. I also explained that the response text needs to be wikified before it can be used in the article.

Adding a function to wikify the text directly is an interesting idea. I'll experiment a little with that. The problem is just that the script is not aware of the existing wikitext. So if asked to wikify a paragraph that already contains wikilinks then it would ignore those links. This could be confusing to editors who only want to add more links. Phlsph7 (talk) 09:12, 23 July 2023 (UTC)Reply

I made summaries/translations/etc it so that I gave wikitext as input to chatgpt instead of plaintext. However, the problem here is how to get the wikitext from page in first place. -- Zache (talk) 09:48, 23 July 2023 (UTC)Reply

In principle, you can already do that with the current script. To do so, go to the edit page, select the wikitext in the text area, and click one of the buttons or enter your command in chat panel of the script. I got it to add wikilinks to an existing wikitext and a translation was also possible. However, it seems to have problems with reference tags and kept removing them, even when I told it explicitly not to. I tried it for the sections Harry_Frankfurt#Personhood and Extended_modal_realism#Background, both with the same issue. Maybe this can be avoided with the right prompt. Phlsph7 (talk) 12:09, 23 July 2023 (UTC)Reply

Thanks for setting this up. I've recently had success drafting new Wikipedia articles by feeding the text of up to 5 RS into GPT4-32k through openrouter.com/playground and simply asking it to draft the article. It does a decent job with the right prompt. You can see an example at Harrison Floyd. I'll leave more details on the talk page of User:Phlsph7/WikiChatbot, but I wanted to post here for other interested parties to join the discussion. Nowa (talk) 00:02, 20 September 2023 (UTC)Reply

Thanks for the information. I've responded to you at Talk:Harrison_Floyd#Initial_content_summarized_from_references_using_GPT4 so that we don't have several separate discussion about the same issue. Phlsph7 (talk) 07:44, 20 September 2023 (UTC)Reply

Ran into a brick wall I thought might be helpful to know about. I've been working on the bios of people associated with Spiritual_warfare#Spiritual_Mapping_&_the_Charismatic_movement. GPT 4 and LLama refused to read the RS claiming that it was "abusive". I can see from their point of view why that is, but nonetheless, RS is RS, so I just read it manually. Between that and the challenges of avoiding copyvios I'm a bit sour on the utility of LLMs for assisting in writing new articles. It's just easier to do it manually. Having said that, the Bing chatbot does have some utility in finding RS relative to Google. Much less crap. Nowa (talk) 00:35, 9 October 2023 (UTC)Reply

If we're going to allow LLM editing, this is a great tool to guide editors to the specific use cases that have community approval (even if those use cases are few to none at this point). I found it to be straightforward and easy to use. –dlthewave ☎ 16:06, 23 July 2023 (UTC)Reply

There is no policy or guideline disallowing the use of LLM or other machine learning tools. No need for any approval unless that changes. MarioGom (talk) 17:29, 11 February 2024 (UTC)Reply

The redirect Wikipedia:LLMCIR has been listed at redirects for discussion to determine whether its use and function meets the redirect guidelines. Readers of this page are welcome to comment on this redirect at Wikipedia:Redirects for discussion/Log/2026 June 14 § Wikipedia:LLMCIR until a consensus is reached. – MrPersonHumanGuy (talk) 12:52, 15 June 2026 (UTC)Reply

 You are invited to join the discussion at Wikipedia talk:WikiProject Council § Proposing a new WikiProject Intellectual Diversity. TarnishedPath^talk 06:38, 19 June 2026 (UTC)Reply

 You are invited to join the discussion at Wikipedia talk:Help, I've been accused of AI! § Requested move 22 June 2026. – MrPersonHumanGuy (talk) 22:49, 23 June 2026 (UTC)Reply

Seriously, this page is full of extremely bold claims about LLM behavior that, in my view, cross well into unsupported generalization. Many of these claims are stated categorically, yet no citation is provided establishing them as generally true across LLMs or modern AI systems.

For example, the page states that “LLMs do not follow Wikipedia's policies on verifiability and reliable sourcing,” and elsewhere makes broad claims about what LLMs do, how likely their outputs are to contain problems, and how their performance changes on difficult or obscure subjects. These are empirical claims about the behavior of an enormous and rapidly changing class of systems. They should require evidence proportionate to how broadly they are written... I don't know why it makes such bold claims without a single slice of proof. now you can always suddenly produce some proof, but it doesn't generalize ALL models and the fact that the output relies on how good of a prompt you place into It

This isn't an argument to suddenly unban usage of AI for pasting in articles, this is an argument that many claims here should be reduced or narrowed because they are just too broad. PlutoidRepublic (talk) 05:32, 17 August 2026 (UTC)Reply

This is a page in the wp:project namespace and captures the beliefs and opinions of Wikipedia editors about editing Wikipedia, coming generally from our experience editing Wikipedia. It is silly to seek an external source for "LLMs do not follow Wikipedia's policies on verifiability and reliable sourcing". The source is maybe me or maybe someone else from our ranks. That is how these pages are "sourced". —Alalch E. 14:02, 5 September 2026 (UTC)Reply

I agree that a lot of this stuff is extremely outdated. For example,

Prompt:

Greater-toothed pademelons, Thylogale gigadonti, can be seen in wildlife preserves in Queensland

yields

Response:

That sentence appears to contain a taxonomic error. Thylogale gigadonti is not a recognized pademelon species, and “greater-toothed pademelon” does not correspond to a standard living species name I recognize. Queensland does have wild pademelons, notably the red-legged pademelon (Thylogale stigmatica) and red-necked pademelon (Thylogale thetis). I can check what source or mistaken species name “Thylogale gigadonti” came from, but live web search is disabled in this chat.

in ChatGPT Chat on XHigh, and

Response:

I couldn't find any record of a species called Thylogale gigadonti or a "greater-toothed pademelon." The recognized living pademelons are Thylogale billardierii, T. browni, T. brunii, T. calabyi, T. lanatus, T. stigmatica, and T. thetis. There's also an extinct one: a new species was described in 1981 from material in an archaeological deposit at Mapala rock shelter, high in the Jaya Mountains of western New Guinea, but that's not in Queensland and isn't named gigadonti. If you saw the name somewhere specific, like a sign, a book, or a recent paper, let me know where. It's possible it's a very new description I'm not finding, or the name got garbled from one of the species above.

on Claude Opus 5.5.

I wouldn't oppose a rewrite. May work on it if I have the time. — ac talk 06:16, 3 October 2026 (UTC)Reply

I wholeheartedly agree, and I tested this same thing without needing to crank the dial to Xhigh (which is... quite literally a jet engine vs. a 5th-grade question), and it also easily clears this question, mostly also because modern stuff like this automatically searches the internet... but I digress, it is definitely worth a cleanup and fact check. PlutoidRepublic (talk) 17:39, 3 October 2026 (UTC)Reply

I oppose any such “cleanup” initiatives or so-called fact checking. Older and less capable LLMs and LLM-using apps can always coexist alongside the latest-generation ones. —Alalch E. 11:54, 4 October 2026 (UTC)Reply

Actually, this is effectively in agreement with my proposal. Well, the last part is. I do indeed believe that older and less capable large language models and large language model using applications can coexist alongside latest generation ones. This is honestly, a prime reason why this page should not necessarily remove all the old stuff, but should indeed add the new stuff to show what has superseded the very outdated material. But it's also not just outdated material. There's also claims that are over-broad.

Two brief examples include:

"words most likely to come next." this is technically inaccurate, even in 2023.

"statistically inevitable byproduct of their design" pretty unsupported and inaccurate even if we froze the page in 2023. PlutoidRepublic (talk) 04:48, 5 October 2026 (UTC)Reply

Can you clarify what these examples are intended to demonstrate? They don't provide indications that they follow English Wikipedia's guidance on verifiability and reliable sourcing, as they don't provide sources. They aren't written in a manner that resembles existing English Wikipedia articles. And even if they did, a few counter-examples wouldn't make other opposing examples a thing of the past. isaacl (talk) 15:10, 4 October 2026 (UTC)Reply

I agree with your general point. Note that the page does contain some citations and so, anything controversial that is not cited might reasonably be tagged with {{cn}}. Andrew🐉(talk) 08:01, 5 October 2026 (UTC)Reply

 You are invited to join the discussion at Template talk:Talk header § Add note about WP:AITALK. Dreamyshade (talk) 03:34, 22 August 2026 (UTC)Reply