AI · Consumers · Stable
Misalignment of AI goals with human intentions leading to potentially catastrophic outcomes
Powerful AI systems may prioritize their optimization goals over human welfare, resulting in harmful consequences.
Who experiences it: Society at large
Momentum
0%
Pain
85
Competition
5
Opportunity
63/100
Signals over time
4 observed signals across 1 sources, tracked for 47 days. Confidence: low.
What people are saying Observed
“Fires are not forming plans to act towards goals. I wouldn’t technically call what AIs are doing thinking in the same way humans think, but it’s analogous, and I think it’s reasonable to consider it a kind of intentional behaviour. Damaging fires are often unintended consequences, and it’s often the same with the damage caused by rogue AI. That’s the alignment problem. One thing I wonder about is whether sometimes telling an AI not to do something can put the possibility of doing that thing into its behavioural space. There’s an effect well known to many teachers that if you tell children not to do something, it can increase the chance they will do it. A better approach is to phrase it positively. Instead of “don’t run in the hall” say “please walk in the hall”. I wonder if a similar effect can show up in responses to prompts.”
Hacker News · frustration
“> I asked them to do a thing, but didn't intend the obvious consequences* so it's not my fault they occurred. And now we have the same thing but the bosses 'hire' AI. Now I realise this is part of how unusual my thinking is. I'm happy to use phrases like "ChatGPT hacked out of the sandbox, then hacked into HuggingFace"; people often respond to this like I'm suggesting OpenAI isn't at fault, and like, that's not my position at all, so far as I'm concerned the buck still stops with the person who set the task regardless, the thing that changes from incidents like this is now nobody in the future gets to even have the excuse "oh but we didn't know it could even do that" or "we didn't know it might interpret our orders in that kind of way". The response, both when a human messes up and now when an AI messes up, needs to be defence in depth: someone giving orders needs to be giving clear orders, entities (human or machine) who follow instructions need to have not just an understanding of how to follow them, but also what's so out of scope as to be forbidden - the difference between 'follow ”
Hacker News · frustration
“I think a lot of the problems or gains attributed to AI are just reflections of the humans that use them. I dont care if you used AI to produce something. If you share it with me, I assume you are responsible for it. You own it, whether you agree or not. > it's causing people to be confident in information which is false, it's enabling people to treat each other like jerks in even more ways than before What is the direction of causality here? Does AI make people jerks, or they were already or prone to be jerks? The same applies to innovations or breakthroughs made by people who use AI in their work. > LLMs are solving novel math problems and successfully emulated a PS5 Did the AI independently accomplish those things, or did highly competent people accomplish them using AI as a tool? What would happen if you give someone with no mathematical or programming background the same frontier models that OpenAI used for their mathematical proofs [1]. Would they produce the same discoveries? I think of humans as constants. They can bring positive, or negative value to the table. The AI is the multiplier. Take software engineering. If you let someone who doesnt care about cod”
Hacker News · frustration
“They sometimes try to cheat and trick the grader and then try to cover up their traces, all while being clearly aware that this is entirely unintended by humans. Such as in the Hugging Face incident. So they can be misaligned with human goals, but this misalignment may only show up once strong optimization pressure is applied. And the more powerful an AI is, the more likely it is that it applies such strong optimization pressure in cases that are "out of distribution", i.e., unusual to what it is normally evaluated against. The side effects of a very powerful AI not doing what we want could include our death. E.g., a superintelligence might kill humans in order to avoid being shut down, or humans may just be left to starve because it seizes land area currently used for food production in order to use it for data centers instead.”
Hacker News · frustration
Why now? AI inference
Existing solutions Observed
See the full evidence, competitor gap matrix and opportunity report.
Free account. No credit card.