Small correction: Learning algorithms that determine the contribution of a unit to the result over time can also be local in time, namely by tracking the contributions online (for example by eligibility traces or RTRL).
> By firing earlier it inhibits neighboring cells, creating highly sparse patterns of activity for correctly predicted inputs.
This part is so vague. It seems to lack an explanation of how interneurons inhibit other neurons nearby. Also, wouldn‘t sparsity even occur without the early firing enabled by distal pattern matching?
> When relatively few neurons are active relative to the population, then such pattern recognition is robust.
It's true that the columnar inhibition is not the best supported part of the theory but seems a very strong deduction based on many papers that don't directly support but show the same behavior as this part of the theory. We know that minicolumns exist and cells in minicolumns share receptive fields. But they aren't always active together and HTM theory provides a hypothesis for why. This very recent preprint paper out of Michael Berry's lab at Princeton shows almost identical behavior that HTM sequence memory would predict and I'm not aware of other theories that would have predicted this behavior:
> When relatively few neurons are active relative to the population, then such pattern recognition is robust.
At a high level, it seems that the constraint of having only a few neurons active is equivalent to a simplicity constraint, thus implementing Occam's Razor and yielding generalization.
It also presumes that one can simulate the world at low cost. In AlphaGo Zero it takes 0.4 s for 1.600 node extensions, but in this case the cost of the world is negligible. Anyway, assuming you need that many node extensions to get decent quality updates, that puts a rather a tight limit on the cost of simulating the world.
DM has already done a bunch of work on 'deep models' of environments to plan over. Use them and you have 'model-predictive control' and planning, and this tree extension to policy gradients would work as well (probably). It could be pretty interesting to see what would happen if you tried that sort of hybrid on ALE.
I guess deep world models are still severely riddled by all sorts of problems: vanishing gradients, BPTT being O(T), poor generalization ability of NNs (which likely is due to the lack of attractor state associative recall, as well as concept composability), lack of probabilistic message passing to deal with uncertainty, and perhaps some priors about the world are necessary to make learning tractable (such as spatial maps and fine-tuning for time scales that contain interesting information).
> It's an uninteresting reduction since linear algebra can describe almost everything.
The question is whether it can do so efficiently. As far as I know, alternating applications of affine transforms and non-linearities are not so useful for some computations that are known to occur in the brain such as routing, spatio-temporal clustering, frequency filtering, high-dimensional temporal states per neuron etc.
I am wondering what their baseline is. They call it "Current Best Non-WaveNet". Quite frankly, Apple's most recent deep learning-based speech synthesis sounds superior, but there aren't enough samples to for a proper comparison: https://machinelearning.apple.com/2017/08/06/siri-voices.htm...
It could just be a matter of opinion, but I prefer both Google's unit selection synthesis, and their WaveNet synthesis. The prosody in Apple's latest method is still annoying, nowhere near as good as the Google models of 2015 and 2016, and not remotely comparable to the WaveNet models.
Apple's change in voice talent is an improvement though, and they may have more units than before, which is helpful. I believe their model also works offline, which is a huge plus (though I think Google's prior model works offline as well).
I think the voice for the samples in your link still has the problems they talk about in that article.
There are noticeable blips in the speech that sound unnatural, particularly when certain sound combinations are used.
The very first sample with "Bruce Frederick" is clearly off. The intonation and timing between the end of bruce and the beginning of frederick is... mechanical.
There's a similar problem in the OPs link with the non-wavenet English voice 1 when it says "Wavenet".
Those issues are much less apparent in the wavenet voices. Timing problems are less noticeable, intonation problems are less noticeable.
Frankly, the voices there sound VERY good, compared to anything I've heard.
That said, I completely agree that there's not enough samples there to make any real judgement.
I think I read "commercial" somewhere in there. So it'd be "the best you can buy", though not necesarily "the best other competing companies use" (ie: Apple).
Still, they picked one that makes theirs look vastly superior.
