I'd imagine this is exactly what intercom is doing for their AI. Any ideas why the performance is worse? Maybe intercom didn't have access to forum data?
TBH, I was pretty surprised too. It made me pretty skeptical of off-the-shelf AI apps in general. I now think that most actually effective AI apps will need to be developed in-house, and that “bolting on” AI to existing apps (e.g. Intercom, Salesforce, etc.) won’t work. I think there are a few reasons:
1. A lot of the useful data for answering questions is in our public docs and community forum answers, which Intercom doesn’t have access to. (And we wouldn’t feel comfortable giving them access to our internal Slack anyhow.) For example, we’ve debugged complicated OAuth issues in Slack, and there is a lot of “context” there that is helpful for answering future OAuth questions (but isn’t available to Intercom).
2. Intercom doesn’t allow you to customize prompts or customize context easily. In our case, for a highly technical product, “prompt engineering” allowed us to radically improve answer quality. We could also use chain-of-thought prompting, which Intercom didn’t support. Together these two improvements probably doubled the answer success rate.
3. We needed to integrate with our data warehouse for in-product context. For example, if a customer has an error with a particular product/feature, knowing what plan they’re on, which features they’re using, which feature flags are enabled, etc. is quite helpful.
