I would be curious to know if anyone has tried a hybrid approach where you have ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		blainm on Feb 16, 2024 \| parent \| context \| favorite \| on: LWM – Open LLM with 1M Tokens Context Window I would be curious to know if anyone has tried a hybrid approach where you have a Mamba-like architecture for longer term recall but it's combined with a transformer for short term memory?

logicchains on Feb 16, 2024 | [–]

Yep, https://arxiv.org/abs/2402.04248 tried a Mambaformer which seemed to perform well.

enonimal on Feb 16, 2024 | [–]

maybe a fun karpathy video here...

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact