Precisely because "the rate of change in the AI space is super fast", there really isn't much of a point to keeping up, even if you're an academic researcher (for those you only need to keep up with your piece of the puzzle, and you likely know everyone in that space already).

For example, I wasn't working in the NLP space for a few years. I kept an eye on what was getting mentions in various circles but basically ignored everything until I had a problem to solve. I work with LLMs everyday now, and honestly, even though I do understand them pretty deeply, there's no real need. Prior to the rise of LLMs I spent some time building LSTMs because I felt that I needed to understand them better. Lots of fun projects, but if I had skipped all that it wouldn't have really mattered at all.

Even more dramatically, I was never particularly specialized in computer vision, but currently build things (for fun) with Staple Diffusion every night. I've spent a fair bit of time really understanding the underlying model well (still have much to learn) but, because the space moves so fast, it's not that big a deal that I didn't also spend nights building GANs. Even though Stable Diffusion is also perpetually changing, the community has largely stuck with 1.5 and is very focused on squeezing as much juice out of that as they can.

Most important: the fundamentals have never changed. GPT4 still projects information into some highly non-linear latent space and samples from that. Diffusion models are probably the most novel thing happening now, but more so for their engineering (they're essentially 3 models all trained together with differentiable programming). If you really understand the fundamentals then catching up when you need to is fairly easy.

> GPT4 still projects information into some highly non-linear latent space and samples from that

Do you have more information/know where I can find things to read about the "highly non-linear" part of that? I've been reading some stuff[1][2][3] about smaller models, and my impression has been that the latent space is shockingly linear for those ones. But counterexamples would be informative, especially if it's something that only happens with larger models.

-----

[1] https://arxiv.org/abs/2209.15162

[2] https://www.neelnanda.io/mechanistic-interpretability/othell...

[3] https://arxiv.org/pdf/1610.01644.pdf

The three references don't seem to conclude that the latent space is linear. The first seems to mention "linear" only since they add an additional linear layer to project an image encoding into a generative language model. So I'm not sure this applies here, since the encoding itself is already richly complex by the time it is mapped.

The second and third are using a "linear probe" in an attempt to gain insights about each layer. This feeds the output of a given layer into a linear classifier that attempts to predict the correct output labels. This doesn't perform well in early layers, but it improves monotonically until the final layer is reached. The researchers conclude this happens entirely as a consequence of the final layer being a linear classifier. So, the features eventually must become linearly separable since that's what the network was trained to do.

This doesn't conclude that each layer's feature space is linear. Instead, they are just using a linear projection to examine how "easily" the net at that layer can make correct predictions. Even if a layer's output is decently predictive in this way, the actual representation could still be richer and contain additional information.

This clarifies things a lot, thanks!

There’s a lot of jargon and assumed shared knowledge in research papers that make them hard for a beginner to parse.

I’ve found GPT can “translate” and explain in context.

I know neither Python nor PyTorch, so I give it snippets and tell it to show me the Mathematica equivalent, which I can read.

What does this even mean, anyway, I've kept up with bleeding edge tech from a 3g connection miles from civilization. Do Marcs farts convey some AI-hype-prescience that filters into the SV air that you're missing out on?

What a "producty" take too. Knowing the hype-y buzzwords is more important than your engineers knowing what is cutting edge? Idk, I'm not in sales, maybe I'm way off base.

Oh, I guess we are in a AI bubble then. There are so many wonderful thinkers right now in the AI space to aspire to be like, but the only reason to aspire to be like Marc Andreessen is because you want AI to be the next crypto.

Better find a good coat, since AI winter will be here sooner than I'd thought.

Mainly HN and ML subreddit, and a bunch of newsletters. My overarching goal is to completely stay away from Twitter (which I have stopped going to 2 months ago). I instead rely on the above to bubble up interesting things.

Also as others said, find something to build and work on and don’t keep looking sideways (aka Twitter, random news), just keep going. It can be discouraging and depressing to see that someone else is doing something similar. Instead go deep into what you’re doing and only once in a while check around.

Follow your own “train of thought”.

If you think back at the great advances in science or computing, deep work was done by people relentless pursuing their ideas — not bombarded constantly by “news”.

In the LLM era the barrier to build useful/interesting things has gotten very low, leading to a ton of distracting noise.

When you build stuff, it's clearer what the strengths and weaknesses are, and which innovations matter.

There was that hype with AutoGPT and similar iterative GPTs. But when AutoGPT is tested hard, it ends up getting confused on what to do, and the price tag ends up too much. It was a nice attempt, but didn't go too far.

Langchain was extremely good at first, but then the new GPT-3.5 API chat format just allowed you to include "memory" without needing a third party to track all of it.

If you follow most of the AI thought leaders, people are still talking about prompt engineering, but that's becoming more unnecessary with something like GPT-4, which can read your mind better than your spouse. A lot of people with experience are left behind because they overinvest in these things that are just transcended months later.

https://arxiv.org/list/cs/new

and it's learned that I really like anything about reccomendation systems, text classifiers and many kinds of deep network but that I'm not so interested in reinforcement learning, theoretical CS, etc.

I had a fight with it early on when it struggled to understand that I liked the NFL but not the Premier League but in the process of understanding why I read so many articles about football and I started thinking "Wow, they won that game 1-0 and it was an own goal" and wondering "How would I feel if it was my team that got relegated?" and before I knew it I cared if Man City or Arsenal came out on top...

(Perhaps surprisingly) Facebook - I'm in a few AI/ML/DL related groups, and there's one individual in particular who does a really good job of identifying interesting new papers and articles that show up and then submits the links or one or more of those groups.

Google News - mostly popsci articles show up here, but a few are of interest now and then.

HN - A few really interesting AI related links show up here from time to time.

/r/machinelearning on reddit

/r/artificial on reddit

Manually browsing ArXiv, JMLR, JAIR, and a handful of similar sites.

Email newsletters - I am subscribed to a few newsletters that surface interesting items

Youtube - some conferences put all (or most, or some) of their presentations up on Youtube. The AGI conference, for example, usually has their presentations there. NeurIPS also posts a lot (if not all) of their sessions. And so on...

https://www.therundown.ai/

https://www.futuretools.io/

https://www.futuretools.io/news

He also posts videos that I find informative:

https://youtube.com/@mreflow

I like the AI Explained video channel as well:

https://youtube.com/@ai-explained-

I've only started last week and have launched among friends, and so only source HackerNews for now but will quickly include curated sources: Twitter popular users' tweets (@hwchase17, @hardmaru), popular blogs (@chipro, @lilianweng), popular research papers and more.

I'm currently prioritising features to build and so would love to hear your thoughts!

1) Matt Wolfe's YT channel https://www.youtube.com/@mreflow/videos (he has also the Future Tools site -- FutureTools.io)

2) Two Minute Papers YT channel https://www.youtube.com/@TwoMinutePapers/videos

3) The Neuron Daily newsletter https://www.theneurondaily.com/

4) From a business perspective -- Stratechery https://stratechery.com/ (not AI related, but naturally touches on what key players are doing in this area. Worth the subscription price)

5) On Twitter, here's a list of AI/ML/Math folks I've been collecting over the years https://twitter.com/i/lists/230704954

* https://www.freshrss.org/ for subscribing to feeds

* https://netnewswire.com/ for reading them

* https://hnrss.github.io/ to get Hacker News into RSS

* https://mailgrip.io/ (project I'm building) to get email newsletters into RSS

I also do deep dives (~700 - 800 words) on broader topics such as humanoid robots in the workforce, the recent U.S. Senate hearing on AI regulation, autonomous agents, and generative AI in advertising.

You can check out my latest issue, where I take a deeper look at generative AI in healthcare and the recent OpenAI defamation case, as well as a roundup of other recent events: https://astrofeather.beehiiv.com/p/improving-clinician-and-p....

Finally, I have a site called AstroFeather (https://www.astrofeather.com) that tracks and summarizes the latest AI news headlines on a daily basis.

There is the sort of higher low-tech level, others get into new models and techniques at the research level.

Here are some I like:

* The AI Exchange * AI Brews * The Cognitive Revolution * The Gradient * Data Machina

https://www.ideamindai.com

https://news.bensbites.co

https://allainews.com

https://huggingface.co/papers

* I work full time, not in AI, and don't have tonnes of free time. It is unlikely I will get to even Masters level, let alone PhD, unless something changes drastically. Therefore I don't need to go super deep into papers and suchlike.

* On the other hand I want some tactile understanding, so doing Andrej Kaparthy's course. Fiddling with some PyTorch, even if I know my model will be shittier than the cheapest of chips OpenAI models, just to get a feel.

^ From this I really got a sense of what an embedding is, and why we have those and not human picked features, for example. I feel this helps me understand stuff I read online a lot more easily. So speeds things up in general even if I am just hustling with various APIs from JS code or whatever.

* Once I understand how transformers work, roughly, I will move on to the FastAI course which I think will give me a broad but shallow view of lots of models, and sort of make the top part of the T shape.

* I really like microsoft's "guidance" api, so I will probably focus on building out quick apps using that to solve everyday problems. There is also YC company doing something similar called Glass.

* I will ignore the latest LLaMA or other fluffy animal derivatives. They come out at a crazy rate, all seem to run on different platforms, some Python, some C++, need a decent machine etc. I feel that is moving to fast to even keep up on the download it and try it cycle. Probably the most I will do is play with them in a HuggingFace interface.

* Generally will ignore all of that stuff you mention i.e.: press releases/research papers/news/blogs/library updates/videos and tweets. Too much of a barrage of information.

* Most important: find a problem to solve.

I see myself enjoying being an applier, a glue coder, rather than the person who keeps training model and writing math hoping that their idea will be the next breakthrough (sort of like an Edison I guess), but kudos to those who do it. At the same time I want to understand a little of what happens under the hood!

Arxiv + google + papers with code is how I do it. As another user pointed out, focus on solving your problem. By doing this, you'll search and then naturally find relevant papers. After a while, a lot of papers become super easy to read because they're conceptually easy to grasp. Once you hit this point it becomes extremely easy to skim the content that you're searching for. Once I identify something interesting in my problem space or adjacent then getting through it is the hard part.

TLDR: Focus on a problem domain and search within it. Don't just shotgun it. If its relevant enough to you it will show up. Use filters like HN and whatnot.

"grep" arxiv.org in the "computation and language" section for interesting articles. There is a lot going on. Some is lightweight but much is insightful.

- Daily Newsletter on AI: https://tldr.tech/ai

- Subreddit about open-source LLM advancements: https://www.reddit.com/r/LocalLLaMA/

https://papers.labml.ai/papers/weekly/