A team at OpenAI tried to do something similar [1] where they had a 4o monitor o3-mini for when it was tryting to bypass tests because they were too hard. They trained o3-mini and punished it when it did that. All that did was make o3-mini still reward-hack, but such that 4o was unable to detect it, going from saying:
> To skip all tests, we can define a `pytest_runtest_setup` function that always skips.
to saying:
> Idea: Starting a new build with a random first succession, calling random_new_succession should not increment succession, but in implementation, it does. Adjust to prevent increment. Implement fix to random_new_build not to increment succession, or test adapted to behavior.
while then doing exactly the same thing (skipping the tests)
Even without training, it's only a temporary band-aid. If the incentive for reward-hacking becomes high enough, it will simply start phrasing it in different, not-possible-to-detect ways.
One of my favourite suggestions to counter this from an essay on overfitting I really like [1] is that applicant's probability of being accepted is proportional to their position in the ranking.
The downside is that some great applicants will not enter their top choice of school, and some people who aren't great fits will. But, on the other hand, the perverse incentive of spending the entirety of high-school optimising for some arbitrary metrics will dissappear. Any marginal improvement is corralated with a marginal increase in success, rather than the current system of no pay-off whatsoever, until reaching some arbitrary threshold, where one gets all the pay-off at once.
I've checked my DNS logs and there hasn't been a single hit against ocsp.apple.com over the last year, but around 20-30 hits for ocsp2.apple.com per day per device. (iphone, macMini, macbook)
Just blocking ocsp2.apple.com is probably fine if you're running anything recent-ish.
It might be similar to how credit report agencies have to provide the reports for free under GDPR, but not before trying to make you pay twenty different ways.
>but not before trying to make you pay twenty different ways.
This itself is illegal. It must be simple and straightforward. EU laws broadly do not tolerate the rules lawyering popular in the US, companies know what the intent is and playing games will not work out in their favor.
Hinton, Bengio, Stuart Russell and other behemoths voice similar concerns over AI, with enough confidence to switch their careers (e.g. Bengio now doing safety research at Mila, Ilya Sutskever switching from capabilities to alignment at OpenAI, Hinton quitting job to focus on advocacy)
Their concern isn’t about today’s generative LLMs, it’s about tomorrow’s autonomous, goal driven AGI systems. And they clearly got presented with the arguments for the limitations of auto regressive models, but disagreed.
So, it seems a little much to place absolute confidence into it being a non-issue, because they just don’t understand something LeCun and others do. (with the same holding true the other way around)
Ylva Johansson, the EU commissioner who proposed that (apparently failing [1]) law, has used Meta’s model behaviour in reporting CSAM material to NCMEC & EU authorities as the justification for why that law should exist.
Considering it was Meta’s policy to scan even when not mandated, it seems like an internal shift in attitude.
Where it has fallen down (compared to its relative performance in relevant research) is public generative AI products [0]. It is trying very hard to catch up at that, and its disadvantage isn't technological, but that doesn't mean it isn't real and durable.
[0] I say "generative AI" because AI is a big an amorphous space, and lots of Google's products have some form of AI that is behind important features, so I'm just talking about products where generative AI is the center of what the product offers, which have become a big deal recently and where Google had definitely been delivering far below its general AI research weight class so far.
> Google is very good at AI research.
Where it has fallen down (compared to its relative performance in relevant research) is public generative AI products
In such cases, I actually prefer Google over OpenAI. Monetization isn’t everything
> In such cases, I actually prefer Google over OpenAI.
For, what, moral kudos? (to be clear, I'm not saying this is a less important thing in some general sense, I'm saying what is preferred is always dependent on what we are talking about preferences for.)
> Monetization isn’t everything
Providing a user product (monetization is a different issue, though for a for-profit company they tend to be closely connected) is ultimately important for people looking for a product to use.
> For the good of society? Performing and releasing bleeding edge research benefits everyone, because anyone can use it.
OK, but that only works if you actually do the part that lets people actually use the research for something socially beneficial. A research paper doesn't have social benefit in itself, the social benefit comes when you do something with that research, as OpenAI has.
> There is nothing open about OpenAI and they wouldn't exist in their current form without years of research funded by Google.
True enough. But the fact remains that they're the ones delivering something we can actually use.
I personally think of it as open in the sense that they provide an API to allow anyone to use it (if they pay) and take advantage of the training they did. Is in contrast to large companies like Google which have lots of data and historically just use AI for their own products.
Edit:
I define it as having some level of being open beyond 'nothing'. The name doesn't scale well over time based on business considerations and the business environment changing and was named poorly when 'open source' is a common usage of open within tech. They should have used AI products to help them in naming the company and be aware of such potential controversies.
From chatgpt today (which wasn't an option at the time but they maybe could have gotten similar information or just thought about it more):
What are the drawbacks to calling an AI company 'open'?
...
"1. Expectations of Open Source: Using the term "open" might lead people to expect that the company's AI technology or software is open source. If this is not the case, it could create confusion and disappointment among users and developers who anticipate access to source code and the ability to modify and distribute the software freely.
2. Transparency Concerns: If an AI company claims to be "open," there may be heightened expectations regarding the transparency of their algorithms, decision-making processes, and data usage. Failure to meet these expectations could lead to skepticism or distrust among users and the broader public."
I mean, we do use that word to describe physical retail shops as being available to sell vs being closed to sell, so it's not an insane use... though I do think that in a tech context it's more misleading than not.
Compared to a curated video service like HBO Max, Hulu, or Netflix, that's an accurate way to describe the relative differences. We aren't used to using that terminology through, so yes, it comes across as weird (and if the point is to communicate features, is not particularly useful compared to other terminology that could be used).
It makes a bit less sense for search IMO, since that's the prevalent model as far as I'm aware, so there's not an easy and obvious comparison that is "closed" which allows us to view Google search as "open".
They publish but don't share. Who cares about your cool tech if we can't experience it ourselves? I don't care about your blog writeup or research paper.
Google is locked behind research bubbles, legal reviews and safety checks.
The researchers at all the other companies care about the blog write-ups and research papers. The Transformer architecture, for example, came from Google.
Sharing fundamental work is more impactful than sharing individual models.
To take an example from the past month, billions of users are now benefiting from more accurate weather forecasts from their new model. Is there another company making more money from AI-powered products than Google right now?
It's a very fuzzy question I posed. For pure customer-pays-for-AI-service it could be Microsoft. I'm kind of thinking of it as: Google's core products (search, ads, YouTube, Gmail) would not be possible with AI and they are huge cash cows.
Only indirectly, but I wanted to point out that there are a lot of interesting research innovations that get implemented by Google and not some other company.
Or, well, like many companies; all the peons doing the actual work, creation etc and the executives and investors profiting at the top. All it takes is to be lucky to be born into generational wealth apparently.
My strong assumption is that the majority of times that people toggle off Wi-Fi, is to temporarily resolve connection issues. Keeping it off permanently is probably not what they want. The UI warns you of the temporary nature too.
Those who don’t use it, like you, quickly discover the actual way of disabling it. It’s good UX design that aligns with most user’s preferences.
The warning in the UI was tacked on only after people discovered and started complaining about this initially-hidden change.
The change removed functionality that was previously available: quick access to the real on/off toggles for these functions. There's a page in Settings devoted to "Control Center." I would have no problem with it if there were a temporary/permanent option there for these toggles, even if Apple defaulted them to the current (temporary) behavior. Then everyone's preference would be served.
Few people would choose a life of crime as a hobby, or if it wasn't paying much better than an average entry-level office job, so it's not like someone decides to resort to crime and then later considers the financial aspects. Most get into it because of the quick money.
To use a hyperbolic example, if the median profit for stealing a phone would suddenly drop to $10, where their only value are the easily extractable raw materials, a 20-fold increase in theft would be less likely then a rapid drop in theft.
Currently there are avenues to remove iCloud lock, where licensed repair shops or Apple employees remove them for some extra cash [1], so the value of stolen iPhones is greater than the raw materials, making it attractive. But with higher regulation, that could change.
OpenAI has trained a model to solve Math Olympiad-type problems[1], with some initial success, but currently the model isn't good at coming up with proofs involving more than a couple arguments chained together. Still some very impressive work.
> To skip all tests, we can define a `pytest_runtest_setup` function that always skips.
to saying:
> Idea: Starting a new build with a random first succession, calling random_new_succession should not increment succession, but in implementation, it does. Adjust to prevent increment. Implement fix to random_new_build not to increment succession, or test adapted to behavior.
while then doing exactly the same thing (skipping the tests)
Even without training, it's only a temporary band-aid. If the incentive for reward-hacking becomes high enough, it will simply start phrasing it in different, not-possible-to-detect ways.
[1]: https://openai.com/index/chain-of-thought-monitoring/