(Translated by https://www.hiragana.jp/)
Bots Are Better than Humans at Solving CAPTCHAs - Schneier on Security

Bots Are Better than Humans at Solving CAPTCHAs

Interesting research: “An Empirical Study & Evaluation of Modern CAPTCHAs“:

Abstract: For nearly two decades, CAPTCHAS have been widely used as a means of protection against bots. Throughout the years, as their use grew, techniques to defeat or bypass CAPTCHAS have continued to improve. Meanwhile, CAPTCHAS have also evolved in terms of sophistication and diversity, becoming increasingly difficult to solve for both bots (machines) and humans. Given this long-standing and still-ongoing arms race, it is critical to investigate how long it takes legitimate users to solve modern CAPTCHAS, and how they are perceived by those users.

In this work, we explore CAPTCHAS in the wild by evaluating users’ solving performance and perceptions of unmodified currently-deployed CAPTCHAS. We obtain this data through manual inspection of popular websites and user studies in which 1, 400 participants collectively solved 14, 000 CAPTCHAS. Results show significant differences between the most popular types of CAPTCHAS: surprisingly, solving time and user perception are not always correlated. We performed a comparative study to investigate the effect of experimental context ­ specifically the difference between solving CAPTCHAS directly versus solving them as part of a more natural task, such as account creation. Whilst there were several potential confounding factors, our results show that experimental context could have an impact on this task, and must be taken into account in future CAPTCHA studies. Finally, we investigate CAPTCHA-induced user task abandonment by analyzing participants who start and do not complete the task.

Slashdot thread.

And let’s all rewatch this great ad from 2022.

Tags: , ,

Posted on August 18, 2023 at 7:04 AM • 21 Comments

Comments

bl5q sw5nN • August 18, 2023 9:47 AM

Humans using robot tool developed by humans do better than humans not using robot tool.

Triumph of the Script Kiddies

Winter • August 18, 2023 9:57 AM

Just get adequate Captcha’s

There are some simple logical puzzles AIs are still failing at

‘https://www.nature.com/articles/d41586-023-02361-7

In a test consisting of a series of brightly coloured blocks arranged on a screen, most people can spot the connecting patterns. But GPT-4, the most advanced version of the AI system behind the chatbot ChatGPT and the search engine Bing, gets barely one-third of the puzzles right in one category of patterns and as little as 3% correct in another, according to a report by researchers this May.

Morley • August 18, 2023 10:45 AM

This race is pretty fun to speculate about. Maybe have a web of trust for people.

I just hope it doesn’t end with corporations or governments the only ones who can prove I’m not a bot.

one-old-conservative • August 18, 2023 12:06 PM

Good. Maybe if bots manage to decimate Captcha, it will disappear! I absolutely have to access a site before I do those !@#$ Captcha things. They are an abomination and need to just go away!

Clive Robinson • August 18, 2023 12:55 PM

@ My Probation Officer…, ALL,

Re : The odds are not a deterrent.

“I can’t imagine myself participating in any competition against say a 100 or 500 people on the other side, of course, one can always try but the outcome in nearly all, if not in every single case will be pretty obvious”

Actually people play those odds or a lot worse many times every day, every day, and they almost always get away…

Yes I know your gut is telling you I am wrong, but have a think about malware developers and those that deploy it. Likewise all the online fraud/theft.

It’s many billions a month every month in total but so little gets reported or logged by authorities let alone investigated that we realy don’t know how much… As somebody pointed out blockchain and crypto-coins gives a very low side estimate…

Now consider the number of “common convicted criminals” in jail in your local population. Estimates say that the number of citizens not in prison v criminals in prison is somewhere between 750 to 3500 depending on which part of the world you live… The statistics from actuaries say ost petty crooks have to committ crime two or three times a week…

The number of bodies odds makes no difference to the criminals, they just keep going.

DomingoP • August 18, 2023 12:57 PM

@Mexaly:

Sometimes I rebel by doing a CAPTCHA wrong over and over.

“Select all images that contain a fire truck”… and now we know who to blame when a Tesla on autopilot crashes into such a truck.

tim • August 18, 2023 1:27 PM

Captcha’s is one tool in the toolbox to protect public sites from abuse. And they still are an effective one.

Good. Maybe if bots manage to decimate Captcha, it will disappear! I absolutely have to access a site before I do those !@#$ Captcha things. They are an abomination and need to just go away!

Here is a thought “one-old-conservative” – stop re-using passwords. Stop putting your birthyear in your username. Or just stop using the internet. And we wouldn’t need captcha’s to assist in protecting your online accounts.

Brodie • August 18, 2023 1:58 PM

Cookies, not CAPTCHAs — that is the problem. Over and over again on almost every website: “Accept,” “Reject,” “Read more.” Enough!

My Probation Officer and FBI Know Who I Am • August 18, 2023 3:55 PM

Good one @Clive,

see, now we can just add your response to the database and have that aspect covered as well.
That’s the beauty of it, I guess, we’d just be “feeding the monster” or anyone else who’d contribute I guess.

@DomingoP, @Mexaly
I like very much, fun Friday, so now I know whose fault it is (@Mexaly) that lately
I have to do 10 or more captchas just to get to a single site.
But seriously guys, that’s an awesome “revolt” idea.
@Mexaly you’re a creative one! Have a good weekend everyone, I gotta run, lotsa captchas waiting to be “trained” 🤣🤣🤣

Me • August 18, 2023 4:21 PM

I had a CAPTCHA test just two days ago to log my hours at work.

It was freaking grueling!

I had to go through like 6 “click all the X” pages before it would let me say “I worked this week, please pay me.”

Robin • August 19, 2023 3:47 AM

I don’t know how many CAPTCHAs are needed to use the site Hotels.com; the last few times I gave up after too many. I no longer use that site.

I have a feeling that using a VPN (ProtonVPN in fact) seems to increase the number of CAPTCHAs some sites (not necessarily hotels.com) demand to get in. Anyone else?

Ted • August 19, 2023 7:52 AM

@Robin

Re: websites, privacy, and CAPTCHAs

I remember seeing this tweet from Matthew Green earlier this year about CAPTCHAs and search sites:

Not loving the CAPTCHA Google now makes me answer every time I search something with iCloud Relay turned on…

And someone added:

Its not only iCloud Relay. It happens whenever you are using VPN. On the bright side it forces me to keep my VPN on and just use @DuckDuckGo since they don’t care if you use VPN or not.

DomingoP • August 19, 2023 6:14 PM

@Ted:

DuckDuckGo … don’t care if you use VPN or not.

Anyone who wants the address of their official Tor onion service can search for “DuckDuckGo onion” there.

Unfortunately, DuckDuckGo often fails to find things, such as quotes from songs or TV shows, that Google has no trouble with—despite being a shadow of its former self. That only works for those of us allowed to use Google, but switching relays about 10 times can usually get around the CAPTCHA (be aware: Google rewrites all search result links to track what you click on). Brave Search looked promising till it starting showing CAPTCHAs too—ones that can’t be used without enabling Javascript.

vas pup • August 20, 2023 6:29 PM

That’s funny — but AI models don’t get the joke
https://www.sciencedaily.com/releases/2023/07/230731122233.htm

This is extract – see link for more details

“Using hundreds of entries from the New Yorker magazine’s Cartoon Caption Contest as a testbed, researchers challenged AI models and humans with three tasks: matching a joke to a cartoon; identifying a winning caption; and explaining why a winning caption is funny.

Large neural networks, a form of artificial intelligence, can generate thousands of jokes along the lines of “Why did the chicken cross the road?” But do they understand why they’re funny?

Using hundreds of entries from the New Yorker magazine’s Cartoon Caption Contest as a testbed, researchers challenged AI models and humans with three tasks: matching a joke to a cartoon; identifying a winning caption; and explaining why a winning caption is funny.

In all tasks, humans performed demonstrably better than machines, even as AI advances such as ChatGPT have closed the performance gap.

!!!This study revealed a significant gap between AI- and human-level “understanding” of why a cartoon is funny. The best AI performance in a multiple choice test of matching cartoon to caption was only 62% accuracy, far behind humans’ 94% in the same setting. And when it came to comparing human- vs. AI-generated explanations, humans’ were preferred roughly 2-to-1.

While AI might not be able to “understand” humor yet, the authors wrote, it could be a collaborative tool humorists could use to brainstorm ideas.

!!!This work was funded in part by the Defense Advanced Research Projects Agency; AI2; and a Google Focused Research Award.”

Erdem Memisyazici • August 21, 2023 6:57 AM

@Winter said it well.

I suppose for most setups the rewards system has to be built in. Most A.I. has to be told what is good and what is bad so asking it to play a game is a great way to distinguish the output behind the monitor.

If it wasn’t done before it won’t know what to do. Just a little creativity in CAPTCHAs can easily overcome 97% in the visual tasks department alone. Of course someone will sit down and train a new network to defeat those puzzles and you’ll have to alter it again but creativity is limitless.

Grima Squeakersen • August 22, 2023 6:23 PM

I despise CAPTCHAS of all kinds, but the most ludicrous examples weren’t mentioned. For ReCAPTCHA, it is evidently at the discretion of the site operator whether any of the identification images are even generated. Quite a number of sites “secure” access by merely requiring a single click, to place a checkmark in a single box, “to prove that you are human”, nothing more than that. NTM that the box and the attendent images are always identical, and always appear at the identical relative screen coordinates. What a colassal waste of bits and time. I haven’t written any serious code in 27 years, but I’d bet I could whip up a widget to automatically answer that challenge in less than 2 days (day 1 would be spent studying the syntax of a modern language or code generator). If I was forced to choose a CAPTCHA that I prefer, I’d take sliding the jigsaw puzzle piece into position version. It’s most likely no less secure than the others, and it is quick to complete.

jaro • August 23, 2023 3:21 AM

Trying to fight against the bots on our forum, I pretty much appreciate that we have at least CAPTCHA tools that disable hundreds of new fake IDs every day that just want to spam.

Still, they sometimes manage, so we have to keep the CAPTCHA tools up to date – and the single check box won’t do at all any more. Unfortunately.

runedust • September 18, 2023 1:57 AM

Check out the book Qualityland by Marc-Uwe Kling. I absolutely love it.
In it, you are a human when you solve a CAPTCHA wrong. If the CAPTCHA is solved correctly it’s a prove that it was a machine 🙂

Anonymous • March 21, 2024 8:15 AM

I just realized that the new captcha where you have to select images with certain objects in them is in fact unpaid labor for AI Training.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.