Part of the cleverness of GANs was to have found a way to train a neural network that generates data without explicitly modeling the probability density.
In a stats textbook, when you know that your training data comes from a normal distribution, you can maximize the MLE wrt the parameters, and then use that for sampling. That's basic theory.
In practice, it was very hard to learn a good pdf for experimental data when you had a training set of images. GANs provided a way to bypass this.
Of course, people could have said "hey let's generate samples without maximizing a loglikelihood first", but they didn't know how to do it properly, how to train the network in any other way besides minimizing cross-entropy (which is equivalent to maximizing loglikelihood).
Then GANs actually provided a new loss function that could be trained. Total paradigm shift!
I'm on board with all of this, I think even before GANs it was becoming popular to optimize loss that wasn't necessarily a log likelihood.
But I'm confused by the usage of the phrase generative model, which I took to always mean a probabilistic model of the joint that can be sampled over. I get that GANs generate data samples, but it seems different.
This is the problem when people use technical terms loosely and interchangeably with their English definitions. Generative model classifiers are precisely as you describe. They model a joint distribution that one can sample.
GANs cannot even fit this definition because it is not a classifier. It is composed of a generator and a discriminator. The discriminator is a discriminative classifier. The generator is, well, a generator. It has nothing to do with generative model classifiers. Then you get some variation of neural network generator > model that generates > generative model. This leads to confusion.
Now, our model also describes a distribution p^θ(x)\hat{p}_{\theta}(x)p^ θ (x) (green) that is defined implicitly by taking points from a unit Gaussian distribution (red) and mapping them through a (deterministic) neural network — our generative model (yellow). Our network is a function with parameters θ\thetaθ, and tweaking these parameters will tweak the generated distribution of images. Our goal then is to find parameters θ\thetaθ that produce a distribution that closely matches the true data distribution (for example, by having a small KL divergence loss). Therefore, you can imagine the green distribution starting out random and then the training process iteratively changing the parameters θ\thetaθ to stretch and squeeze it to better match the blue distribution.
This is precisely a generative model in the probabilistic sense. The section on VAEs spells this out even more explicitly:
For example, Variational Autoencoders allow us to perform both learning and efficient Bayesian inference in sophisticated probabilistic graphical models with latent variables (e.g. see DRAW, or Attend Infer Repeat for hints of recent relatively complex models).
The issue with GANs is that - while they model the joint probability of the input space - they aren't (easily) inspectable in the sense you can't get any understanding of how inputs relate to outputs. This means they appear different to traditional generative models where this is usually a goal.
They are reasonably competitive with GANs. I haven't kept up on the latest models on either side, but VAEs have historically tended to be a little blurrier than GANs.
I think VAE's haven't been the state of the art since around 2016-2017? They have been squeezed from both directions, autoregressive models on the compression side, GAN's on the generation side.
They are still fairly competitive on both sides though.
Yeah, I guess I was thinking of VQVAE as a state-of-the-art example, but it was indeed 2017. Time flies! It's still pretty influential on newer systems though, e.g. OpenAI's DALL-E that made waves earlier this year has a VAE component (in addition to a Transformer component).
I also found that the best system is having the first layer of folder organization be "which period of my time is this from?".
Conceptually, it's easier to think of "music from high school" than about the specific mix of subgenres from my playlist back then. Same for documents that I saved. Those ICQ logs from high school are there. They don't belong in the same folder as the stuff I wrote yesterday, even though they could be of a similar nature.
You're setting a very high bar there, and then claiming that losing access to your gmail account isn't worse than that therefore it's not life changing.
Email ends up being the form of online identity for a lot of people, myself included, so that almost every service that I sign for has my email address as ID. If that email address isn't the ID, it's the preferred way of resetting passwords. I wouldn't be super happy about Facebook being my online ID, nor my cell phone number (see SIM swapping problems).
It's life changing in the same way that losing all your personal documents in a fire sets you up accounting nightmares. Moreover, you're making very light a situation about losing all your pictures. I'm not talking about food pictures, but there's plenty of "me" that's contained in being able to look at pictures of important events of my life (which is why I don't rely only on cloud backups for that).
I do have a lot of stuff that I’d be sad about if lost on Google. And yes, I would be inconvenienced to contact all the services for an email change. But when talking about how our society got to where it is now, I just can’t see the moral weight of these kinds of monopolies in the context of just losing access.
I'm sorry that that was your PhD experience. It's a pity that enthusiastic students end up there, often because they're not given the right environment to thrive (e.g. a good lab and a supervisor that cares).
There isn't much to say to respond to that, apart that it seems to me that, in a parallel universe, you might have had a more fulfilling experience, or you might have cut your losses and walked away sooner.
That's the cruel aspect of the PhD. It really seems like a lot of important things are outside of one's control, especially when it comes to important factors in mental health. Nobody's starting a PhD with the goal of sinking hours into Reddit and Buffy because they feel awful about their PhD experience.
I met James Randi around 2007 when he came to campus at UBC to give a talk. He told the following story about a magic trick that he worked on for years, which concerned guessing the timing of his death. I haven't heard him tell that story elsewhere, so now seems like a good time to share it.
He said that for a good number of years, every time before going to bed, he would write on a little card that he predicted he would die that night during his sleep. In the morning, he got up and happily threw away the little card. Every day. For many years. His concept was that, on the rare chance that this actually happened to be last day, people would think that he pulled the ultimate magic trick. People would not suspect that he wrote this on a little card every day because, well, nobody does that.
Well, I guess he stopped doing it at some date before 2007. That's why he told an audience of 500 people about it. He sorta "cashed out" by making it a fun story about him being clever, instead of a last magic trick. He must have simply gotten tired of doing this every day and seeing all those cards in the waste basket (or burning them?).
If he had indeed tried to pull that trick in 2020, a lot of people would have remembered that he said he was setting it up.
I have also had bad experiences with "Backup and Sync", which led me to abandon Google Drive right when I was seriously considering ditching Dropbox.
Given Google's reputation to ditch their own products, I guessed this was some side projects that some Googlers did, and it was never in Google's main strategy to allow people to sync their Google Drive to their local machines. Quite the opposite, actually.
My current gripe with Dropbox is that I'd like to basically be able to pay 4x the "Dropbox Plus" cost in order to get 4x the storage (without having to manage 4 separate accounts). Having 2TB isn't enough, but having "infinite" with Dropbox Business certainly is more than I want.
Nobody is going for absolute certainty here. That bar is too high in any conversation.
His point was mostly that, way before you achieve the kind of AGI portrayed in fiction, you'll have semi-intelligent interdependent systems that cause a lot of trouble due (like the kind that already happens to a lesser degree). Those are the ones that we should worry about right now.
I have a friend who was participating often in psychology experiments at Stanford, and he became familiar with the whole procedure of letting the subjects believe that they were interacting with another person via a computer, when in fact they were interacting with a program (makes everything more standard and easier to analyse).
One day he participated in this "split or share" kind of experiment, and he was ruthless. Nobody's emotions would be damaged by acting nasty and never sharing with the computer program.
Turns out, it probably was actually a real person who was behind on the other side. He saw some old woman crying, coming out of some adjacent room after the experiment was over.
So, yeah, different social conventions definitely apply.
Actually setting up a software environment that connects two participants in the same simulation, and have it run without any bugs is usually way beyond the capabilities of the psychology students running the experiment. Not to mention the bother of coordinating the different teamed up participants (“Did you click start? The other participant is waiting for you to click sta— oh no it timed out. Hang on I'll restart the pairing sequence…”).
If you participate in one of those studies on campus, you are facing a computer running a local bit of standalone software, or a webservice running a questionnaire.
Surely you could solve this problem quite simply by having both participants and the experimenter join a group chat using COTS software+accounts (Skype, Hangouts, Whatsapp...) pre-installed on the machines?
You would normally want to limit their input to a few form fields so the data can be analysed, rather than letting them have a conversation. Look at the ultimatum game for example.
There's the reverse where, what if AIs do develop feelings like that? Why shouldn't they be treated with the courtesy we treat other humans?
Just thought of these:
- Would this kind of personal assisstant be useful for people on the autism spectrum, to help navigate the kind of implied and unspoken narratives that people with autism seem to have difficulty with?
- Would there be a call to train Duplex to speak in a way that is more comfortable for people on the spectrum?
- Or any of the neurodivergent tribes for that matter?
I've said this before, but in reality strong AI will be another species. Every species of any moderate intelligence is expected to be treated with some courtesy and respect, but their social norms are far different from that of a human and we tailor our interactions with them in different ways. If I'm chewing gum and a human sees me, I might offer them a piece too. I'd never do that to a dog, no matter how much the dog wanted me to. If I saw a human chewing on the grass, I might stop them and ask them some questions to see if they're okay or need medical attention. If I saw a rabbit doing it, I might take a picture because it's cute, but I'd leave it alone (unless it was in my vegetable garden).
There is no reason to expect that strong AI will share the exact same feelings we do unless someone explicitly programs it to (which would be a mistake). Any truly emergent emotional behavior on the part of an AI would be very likely to differ substantially from ours. Making a chatbot "sad" is not the same as making a sapient (or even sentient) being sad. If AI ever achieves sentience, we're going to have to learn what makes it uniquely happy and sad.
Sure. There is also the reflection: how we think about and treat AI, or any other sentient species speak as muc about how we understand what it is to be human.
> Why shouldn't they be treated with the courtesy we treat other humans?
Because courtesy requires some amount of empathy and mental effort on top of what's required just to communicate your point. That effort is wasted on a machine.
Sure that's fine for some, but I personally don't see myself bothering to faux-empathize with a machine since I really don't have a mental-model for that kind of entity, so the empathy would be false in any case. Empathy for machines may in fact be cutting straight to the point and not worrying about emotional impacts since maybe machines value efficiency and conserved-cycles.
That mental-modelling would be cognitive empathy. Affective empathy is another, where the empathy is felt, rather than modelled out.
Then there is a particular kind of emotional attitude / posture / intent, where the something is treated as if it were sentient. It is neither cognitive or affective empathy, ao much as a shift in how things are experienced. It isn't something invoked through will. Hard to describe if you've never experienced altered consciousness states.
That particular shift is a fairly important experience for people (or at least for neurotypical), but is not obvious why to anyone who had never experience that shift. (I am speaking about neurotypical people, and some neurodovergent people, not necessarily someonr with a pauci-affective empathy neural architecture, such as people who are psychopaths).
You could even write an article asking why that original article was even written, and it might make for more interesting content.