34

Screenshot / Code Snippet

"Detect OpenAI" link in post menu; notice with results

About

This script adds a link to each post on SE, revision on SE revision pages, and post Markdown views on metasmoke, to check it against Hugging Face's AI detector.

The first click will test the Markdown text using the Hugging Face detector API for the likelihood it was created by GPT and report that as a percentage likelihood that the post is fake.

Lots of users and spammers edit the text provided by ChatGPT a bit, so if you click the button a second time (or use Alt-Click) an iframe to the detector demo page will open with the Markdown text pre-filled for you, so you can try some manipulations to see if the content was edited from ChatGPT source text. For example, on metasmoke it looks like:

Animation of a post on metasmoke. First the base % FAKE is determined of 24.94%, then an iframe is opened showing that after removal of the spammer's added text the detection is 99.86% FAKE

Download

The script can be installed via this link; source code can be found here.

Platform

Tested with Violentmonkey in Firefox on macOS, but it's likely to work on all browsers and platforms. (Please let me know if it doesn't!)

Contact

Post a comment or answer here, or create an issue/pull request on GitHub.

Improve this question
5
  • The detector seems to have stopped working for me, should I create a bug report? It flashes an ellipsis briefly when I click the "Detect OpenAI" link, but then nothing. The Javascript console shows "openai-detector:1 Uncaught SyntaxError: Unexpected token 'Y', "Your space"... is not valid JSON at JSON.parse (<anonymous>) at receiveOpenAIDetectionDataForButton (<anonymous>:62:25) at userscript.html?name=Stack%2520Exchange%252C%2520OpenAI%2520detector.user.js&id=93dddf2d-810f-485c-aadf-f10370b0623e:604:20"
    – tripleee
    Commented Mar 27, 2023 at 8:58
  • 2
    @tripleee right now, the Hugging Face website is temporarily unavailable - openai-openai-detector.hf.space gives "Your space is on error, check its status on hf.co".
    – Glorfindel ♦
    Commented Mar 27, 2023 at 9:01
  • 1
    Heh: Your space is on error ... I had that, too. Ironically, it sounds so much like an AI-generated message. ;-P
    – Adrian Mole
    Commented Mar 28, 2023 at 13:05
  • A very minor complaint; the reverse tab arrow "⭰" (U+2B70 LEFTWARDS TRIANGLE-HEADED ARROW TO BAR) that you're using for the de-indent button doesn't show up for me. "⇤" (U+21E4 LEFTWARDS ARROW TO BAR) would work fine.
    – miken32
    Commented Apr 21, 2023 at 19:09
  • @miken32 fixed in v0.14 - unfortunately, the detector API is down right now (the new version also shows an error message in those cases - I don't know how to show one in the proper style, but I hope someone else can improve that).
    – Glorfindel ♦
    Commented Apr 27, 2023 at 11:56

7 Answers 7

Reset to default
4

Please add the button for new answers as well or posts that have been reloaded in general (because of an edit, for example).

Improve this answer
1
  • 1
    Fixed in version 0.4 - well, at least for edits, and I'm quite sure it works for new answers as well. Please let me know if it doesn't, and thanks for the suggestion!
    – Glorfindel ♦
    Commented Jan 1, 2023 at 17:52
4

Based on this comment I replaced the alert message with a normal message. I created a PR (Pull Request) on GitHub.

New look:

Image: "According to Hugging Face, the chance that this post was generated by OpenAI is 80%."

Tested on:

Tampermonkey v4.18.1

Chrome Version 108.0.5359.125 (Official Build) (64-bit)

Improve this answer
5
  • 6
    Thank you; I thought it would be better to use the standard Stacks notice instead, so I've made an update on my own.
    – Glorfindel ♦
    Commented Dec 25, 2022 at 11:34
  • 3
    @Glorfindel you can use StackExchange.helpers.showToast('message') for simplicity.
    – double-beep
    Commented Dec 25, 2022 at 11:58
  • 3
    @double-beep TIL ... did I miss / forget about some documentation?
    – Glorfindel ♦
    Commented Dec 25, 2022 at 12:41
  • Any documented link for all StackExchange.helpers.--- ??
    – 4b0
    Commented Dec 25, 2022 at 12:50
  • 4
    @Glorfindel it's not documented anywhere. There are types for some SE functions.
    – double-beep
    Commented Dec 25, 2022 at 13:22
3

It seems that comments (and formatting blocks on a post?) containing a link to a post's revision also have a non-working "Detect OpenAI" link appended.

Example:

You can prepend a paragraph with >! (like this) to add a spoiler block, but it only works on questions and answers, not comments :)Detect OpenAI – Aki Tanaka♦ Jan 13 at 15:16

Screenshot

Improve this answer
4
  • Another example: stackapps.com/revisions/16aeccc4-4344-45a7-aeef-ce3391de7ec3/… -->
    – Andrew T.
    Commented Jan 17, 2023 at 15:50
  • 1
    The link has to end in view-source then it gets picked up as a "revision link". It's supposed to only work in the revisions view.
    – VLAZ
    Commented Jan 17, 2023 at 15:54
  • @VLAZ ah, I see. Yeah, I guess this is really an edge case since it's very rare for someone to link to the revision's source.
    – Andrew T.
    Commented Jan 17, 2023 at 16:00
  • 1
    At least for me, using the most recent version of the userscript, the button appended to the end of the comment is functional. While this might be unintended, and the formatting could be better, I can see some situations where this could be quite useful.
    – Makyen
    Commented Jan 17, 2023 at 17:28
3

Looks like this commit broke the script on Tampermonkey on Chrome.
The current version (0.9) doesn't work.

When I click "Detect OpenAI" it throws an error

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'dispatchEvent')

and points to this line.

Improve this answer
3
1

I'm using macOS Ventura, Google Chrome, TamperMonkey

Stack Apps says...

Stack Apps says:

OpenAI Detector encountered a problem getting data from https://openai-openai-detector.hf.space/openai-detector. The browser console may have more information.

enter image description here

ntent.js:8 injected: Refused to connect to "https://openai-openai-detector--5smxg.hf.space/openai-detector?%3C!--%20thumbnail%3A%20https%3A%2F%2Fi.stack.imgur.com%2F8Q61x.png%20--%3E%0A%3C!--%20version%3A%200.1%20--%3E%0A%3C!--%20tag%3A%20%5Bmoderation%5D%20--%3E%0A%3C!--%20excerpt%3A%20Adds%20a%20button%20to%20check%20the%20probability%20that%20a%20post%20was%20written%20by%20a%20bot%20--%3E%0A%0A%23%23%20Screenshot%20%2F%20Code%20Snippet%0A%0A%5B!%5B%22Detect%20OpenAI%22%20link%20in%20post%20menu%3B%20notice%20with%20results%5D%5B1%5D%5D%5B1%5D%0A%0A%23%23%20About%0A%0AThis%20script%20adds%20a%20link%20to%20each%20post%20on%20SE%2C%20revision%20on%20SE%20revision%20pages%2C%20and%20post%20Markdown%20views%20on%20metasmoke%2C%20to%20check%20it%20against%20%5BHugging%20Face%27s%20AI%20detector%5D(https%3A%2F%2Fhuggingface.co%2Fopenai-detector).%0A%0AThe%20first%20click%20will%20test%20the%20Markdown%20text%20using%20the%20Hugging%20Face%20detector%20API%20for%20the%20likelihood%20it%20was%20created%20by%20GPT%20and%20report%20that%20as%20a%20percentage%20likelihood%20that%20the%20post%20is%20fake.%0A%0ALots%20of%20users%20and%20spammers%20edit%20the%20text%20provided%20by%20ChatGPT%20a%20bit%2C%20so%20if%20you%20click%20the%20button%20a%20second%20time%20(or%20use%20%3Ckbd%3EAlt%3C%2Fkbd%3E-%3Ckbd%3EClick%3C%2Fkbd%3E)%20an%20iframe%20to%20the%20detector%20demo%20page%20will%20open%20with%20the%20Markdown%20text%20pre-filled%20for%20you%2C%20so%20you%20can%20try%20some%20manipulations%20to%20see%20if%20the%20content%20was%20edited%20from%20ChatGPT%20source%20text.%20For%20example%2C%20on%20metasmoke%20it%20looks%20like%3A%0A%0A%5B!%5BAnimation%20of%20a%20post%20on%20metasmoke.%20First%20the%20base%20%25%20FAKE%20is%20determined%20of%2024.94%25%2C%20then%20an%20iframe%20is%20opened%20showing%20that%20after%20removal%20of%20the%20spammer%27s%20added%20text%20the%20detection%20is%2099.86%25%20FAKE%5D(https%3A%2F%2Fi.stack.imgur.com%2F5wyoP.gif)%5D(https%3A%2F%2Fi.stack.imgur.com%2F5wyoP.gif)%0A%0A%0A%0A%23%23%23%20Download%0A%0AThe%20script%20can%20be%20installed%20via%20%5Bthis%20link%5D(https%3A%2F%2Fgithub.com%2FGlorfindel83%2FSE-Userscripts%2Fraw%2Fmaster%2Fopenai-detector%2Fopenai-detector.user.js)%3B%20source%20code%20can%20be%20found%20%5Bhere%5D(https%3A%2F%2Fgithub.com%2FGlorfindel83%2FSE-Userscripts%2Fblob%2Fmaster%2Fopenai-detector%2Fopenai-detector.user.js).%0A%0A%23%23%23%20Platform%0A%0ATested%20with%20Violentmonkey%20in%20Firefox%20on%20macOS%2C%20but%20it%27s%20likely%20to%20work%20on%20all%20browsers%20and%20platforms.%20(Please%20let%20me%20know%20if%20it%20doesn%27t!)%0A%0A%23%23%20Contact%0A%0APost%20a%20comment%20or%20answer%20here%2C%20or%20create%20an%20issue%2Fpull%20request%20on%20GitHub.%0A%0A%0A%20%20%5B1%5D%3A%20https%3A%2F%2Fi.stack.imgur.com%2FmDMTr.png": Request was redirected to a not whitelisted URL
Mt @ content.js:8
userscript.html?name=Stack%2520Exchange%252C%2520OpenAI%2520detector.user.js&id=16694645-97d1-4f98-983b-63d22e4f6426:1311 OpenAI Detector error response: Object
(anonymous) @ userscript.html?name=Stack%2520Exchange%252C%2520OpenAI%2520detector.user.js&id=16694645-97d1-4f98-983b-63d22e4f6426:1311
VM90:75 Uncaught TypeError: Cannot read properties of null (reading 'fake_probability')
    at receiveOpenAIDetectionDataForButton (<anonymous>:75:39)
    at userscript.html?name=Stack%2520Exchange%252C%2520OpenAI%2520detector.user.js&id=16694645-97d1-4f98-983b-63d22e4f6426:1277:20

Workaround suggested by script author:

replace

@connect     openai-openai-detector.hf.space

by

@connect     *
Improve this answer
2
  • 1
    Confirmed this still happens today, if we could get this fixed that'd be great.
    – Thomas Ward
    Commented May 8, 2023 at 5:56
  • 1
    @ThomasWard This should be resolved with version 0.15, which is now available..
    – Makyen
    Commented May 10, 2023 at 22:05
1

I'd like to be able to scan only part of a post for OpenAI detection, perhaps via using normal text selection on the post & hitting a scan button.

In that event, the script could use window.getSelection() to get the text to check.

A recent post was almost certainly GPT output—it had all the hallmarks—but the answer was written to lead with a paragraph representing the prompt they gave to the generator. The full answer had 66% confidence. The bit after the prompt was 98.5% confidence once I copied it over to another GPT detector, which is more what I was expecting to see.

Improve this answer
2
  • 1
    If you click "Detect OpenAI" twice (once to run the regular detection, then again) underneath the post you'd get an extra area open up which allows you to edit what text gets checked. Also has some convenience buttons to strip away some formatting but you can also freely edit it and cut out any text you are not interested in.
    – VLAZ
    Commented May 28, 2023 at 17:00
  • @VLAZ Oh hey, I've never noticed that! That's pretty good and will do the trick here.
    – doppelgreener
    Commented May 28, 2023 at 17:03
1

Please make this user script work on election pages.

Currently, the nomination phase of the 2023 Web Applications Moderator Election is running. The link is shown, but apparently, nothing happens.

Snapshot showing Detect OpenAI link

Looking at the web browser's JavaScript console, the following error message is logged:

JavaScript console

Uncaught TypeError: Cannot read properties of undefined (reading 'href')
    at HTMLAnchorElement.handlePostMenuButtonClick (<anonymous>:393:37)
    at HTMLAnchorElement.dispatch (jquery.min.js:3:12444)
    at r.handle (jquery.min.js:3:9173)
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .