Hi all, eng who worked on vectors @ Retool here. We’re excited to get HN’s feedback on a new product I’ve been working on: Retool vectors. There’s been a lot of hype around AI, but as I was thinking about what we could launch, there were really very few AI apps that are today _actually_ delivering business value. For example, there are tons of chatbots, summarization bots, etc., but it wasn’t clear how to use them. There were bots that we could use for answering support tickets (we use Intercom), but when we tried using them, the results were far worse than expected (it only managed to close ~5% of our tickets, and required substantial human intervention).
I then tried building a custom AI chatbot for our support team, powered by a vector database (with previous answers, our docs, and community forum data). In my prototypes, the successful response rate was much higher (around 5 - 7x higher, ~30%). But it was surprisingly hard to build this bot: I needed to stitch together vector databases, custom integrations with Intercom, cronjobs to sync data, etc.
That’s why we built Retool Vectors. The idea is that we want to build a vector database that has full ETL from whatever inputs you have (e.g. Intercom, a postgres database, Salesforce, a community forum, a website, etc.). It’s always kept up to date. (We’re still working on some of these features, but I decided to try and launch this week because I just wanted to get feedback from HN, haha.)
I think the industry has now settled on vector databases as the best way to provide context to LLMs. I hope that Retool Vectors can be a much easier way of getting data into it.
>But it was surprisingly hard to build this bot: I needed to stitch together vector databases, custom integrations with Intercom, cronjobs to sync data, etc.
>That’s why we built Retool Vectors.
Reminds me of Tobi Lütke who founded Shopify like that. When he built an ecommerce store to sell snowboards and realized how hard it was to set that up, he thought it was better to sell ecommerce stores than to sell snowboards
This looks exciting! I was at a shop where we adopted retool to take the burden off of the engineering team for internal tooling, and I think this is taking it to the next level. Looking forward to see more.
Congrats on the launch! Definitely felt the pain of building a custom ETL into a vector db for a recent LLM project. Is there a way to use this independently of the Retool UI builder?
Thanks! Yes so under the hood it's all in a postgres DB with a pgvector column + some metadata. If you go to retool database, you can grab the connection string and use the embeddings with your own apps. I'd like to open this up to bring your own database and integrate with other common vector DBs (Chroma, Pinecone, Mongo vector, etc) next
Would you like to write about the process of creating this for Stack Overflow blog or share some quotes we can use? We're planning to publish a few things on vector DB, embedding, and RAG in the near future.
I'd imagine this is exactly what intercom is doing for their AI. Any ideas why the performance is worse? Maybe intercom didn't have access to forum data?
TBH, I was pretty surprised too. It made me pretty skeptical of off-the-shelf AI apps in general. I now think that most actually effective AI apps will need to be developed in-house, and that “bolting on” AI to existing apps (e.g. Intercom, Salesforce, etc.) won’t work. I think there are a few reasons:
1. A lot of the useful data for answering questions is in our public docs and community forum answers, which Intercom doesn’t have access to. (And we wouldn’t feel comfortable giving them access to our internal Slack anyhow.) For example, we’ve debugged complicated OAuth issues in Slack, and there is a lot of “context” there that is helpful for answering future OAuth questions (but isn’t available to Intercom).
2. Intercom doesn’t allow you to customize prompts or customize context easily. In our case, for a highly technical product, “prompt engineering” allowed us to radically improve answer quality. We could also use chain-of-thought prompting, which Intercom didn’t support. Together these two improvements probably doubled the answer success rate.
3. We needed to integrate with our data warehouse for in-product context. For example, if a customer has an error with a particular product/feature, knowing what plan they’re on, which features they’re using, which feature flags are enabled, etc. is quite helpful.
When I was building our support chatbot powered by a vectorized DB, I couldn’t actually find a project that helped with syncing Intercom chat data, Discourse community forum data, and web crawling data into a vectorized database. Do you have any suggestions? This was really a “scratch my own itch” product (not something to “generate leads for retool sales”), so genuinely asking out of curiosity! (If there are better products maybe we should kill this project and just recommend people use that instead, haha.)
And no, this is definitely not making money. (It’s free after all, so I guess we’re probably losing money on this.) To be honest, as an engineer @ Retool, I work on things because I think they’re cool and could be useful to our customers… not because it has to make my company money.
Not hating, just asking - thanks for the answer! My personal observation is there are a bunch of llm + vectordb solutions, I agree the other integrations are key, but I think we both know they are tractable problems given sufficient motivation. I've also been following some of the push back on vectordb as the best way to solve this kind of problem. Overall, I was hoping to understand what retool stood to gain with this approach and I got the answer :).
My personal opinion (besides this constant flux of Retool product announcements) is that I truly tried and gave Retool a chance... but man, it was tough to get started, I rather just code from scratch with some boilerplate.
Hey I'm on the team at Retool and this is a place we're spending a lot of time right now. We're still working through the challenge of showing the depth of the IDE while balancing the goal of a lot of new users to maybe just get an admin panel or support dashboard that connects to their existing db or salesforce.
I'd love to hear from you (or anyone else reading this) on how your getting started experience went, particularly if you found yourself wanting to just eject into writing a script or app elsewhere. I love to pair on apps whether it's moving existing code into Retool or starting fresh to see the hurdles. My email is my HN username @retool.com
BTW just sent a twitter DM, thanks again for the feedback.
I recently discovered this combination and my mind was blown a bit.
The key part to me is ChatGPT --> Code --> Execute in real world (not sandboxed) --> achieve whatever you want out there via APIs.
I am eager to learn about how this works and maybe look to replicate or push for similar connectivity with other code execution / notebook environments (say Amazon Sagemaker)
If anyone can link me to disccussion groups, active communities or github projects that are toying with these ideas -- I would love to dive in.
Awsome! With Lowdefy we tried to build a low-code framework that works like code. We’ve developed a schema in which to define applications and we’ve built all kinds of apps for enterprise customers. Massive, advanced CRM systems, call centre solutions, ticketing systems, a light MRP, all kinds of survey apps and so many dashboards. Even our docs and our website are Lowdefy apps!
Give Lowdefy a try and reach out it you have any questions or want to see what is possible :) (We need to invest a lot more into content and examples, bootstapping is a grind!)
Founder @ https://airplane.dev here - putting in a shameless plug for Airplane, where you _do_ write code from scratch. :) I think it depends on the builder and use case, and it's not a fit for everything, but our users like / need to manage their tooling as code and are more productive for it.
The people who use retool and never coded will not realize that it would have been much easier if they just learn to code . If they build anything slightly complex, they are essentially programming with the more difficult and slow UI
Would you mind expanding why it was tough to get started with Retool?
We are building https://github.com/trypromptly/LLMStack, a low-code platform to build LLM apps with a goal of making it easy for non-tech people to leverage LLMs in their workflows. Would love to learn about your experience with retool and incorporate some of that feedback into LLMStack.
I love Windmill. I've built some handy internal tools for populating a smoke forecasting app's daily data, some dumb UIs for crud interfaces, and various workers and schedules. I could do it all without windmill, but I actually really like the scaffolding it provides.
I don't understand the appeal of these tools. I've tried it, but it's a complete cognitive overload, having to learn some proprietary UI to do simple things. I don't understand why adding for-loops by pointing, clicking and entering some textbox on the side for the expression is easier than simply typing it out.
It feels like all these "low code" tools are aimed at a tiny audience of technical people who are somehow technical enough to understand these UIs, but not quite technical enough to quickly roll their scripts and prototypes out the "traditional" way.
If you use Windmill's flows, you get a lot of stuff done for you, for instance caching in each step (and in the whole Flow), concurrency, retries, mocking, suspend/approval. You can also mix and match different languages (in one step you can use Python, another you can use TypeScript, etc). Afterwards, you can trigger your script/flow via POSTs, cron, via a auto-generated UI or via a custom UI.
We changed our pricing earlier this year to be a lot more affordable for use cases with high end user counts (https://retool.com/blog/pricing-v2/). Let me know if that works better for you?
If you’re looking for external apps, check out portals (https://retool.com/products/portals) that has custom volume discounts for many, many users
I mean, let's be serious for a second, MS PowerApps, an Enterprise tool every CIO knows by now is potentially cheaper than Retool's unknown Enterprise price: https://powerapps.microsoft.com/en-us/pricing/
There is not much to talk about, I don't see it, maybe Retool is targeting a different ICP. I don't see how a CIO would go for Retool when they can reach for PowerApps which is also potentially cheaper. PowerApps also integrates with the rest of the MS stack.
I am not saying I like it, I prefer Slack over MS Teams too, but Retool is a very hard pitch right now for Enterprise, and I don't see AI driving any points in that discussion.
We've been using the ChatGPT API from Retool Workflows (for spam detection) and it's been valuable for our business. The built-in vector DB looks interesting
How does the spam detection work? Does chat gpt output something that your application can understand, like {isSpam: true}, or does it output a sentence in english?
It outputs both - { isLikelySpam: boolean, reason: string }
Then we have an inbox app (also made in Retool) that our support team uses to manually review any submissions that are isLikelySpam = true. The <reason> helps to understand why it was flagged.
Our use case is for a form builder (https://fillout.com) but I imagine this type of use case is pretty common for any app that has user-generated content
I'm interested in this aspect of llms too, are you simply just passing it some input (email, customer message) and asking chat gpt to decide if it's spam? Do you provide any prior examples of spam / not spam or just rely on the knowledge already embedded within the model?
Spam detection is a classic example for classification problems. I guess I'm trying to gauge whether there's an entire suite of traditional problems that llms solve well enough by simply asking a question of the base model. I've found a few areas in my own work where this is the case.
We give 2-3 examples and find that it works pretty well (few shot fine tuning) but haven't tried actual fine tuning yet so I don't have a 1-1 comparison.
We also have other spam filters that are not LLM-based. One of the main benefits of the LLM-based approach is that it's good at catching people who try to avoid detection (e.g. someone purposefully mis-spelling suspicious words like "pa$$word")
Can someone who's been there tell me how the "Trusted by developers, startups, and Fortune 500s" works? How does Retool (or any company) get another company's logo here?
It’s also very common for a small team (or individual) at a large company to use a tool, and the tool can plaster that logo on their homepage even if the use case was insignificant.
For a lot of companies, logos on homepages is more of an “ask for forgiveness later” type of thing. Not uncommon to do it without permission.
It’s also very common for TOS to include the rights to use your company logo for marketing purposes by default.
For us, every single one of the logos we display on https://retool.com has a committed contract with Retool where they agreed to display their logo. (Generally our champion does need to go ask a VP; in return we offer a discount.) 100% of the logos that we show pay us more than $50k a year. (80% of them pay us more than $100k a year, and some % of them pay us more than $1M a year, hah.) We wouldn't want to display their logo otherwise! (Since it'd be misleading, but also because it'd be problematic — if say — Taco Bell engineer came and asked us who exactly is using Retool at Taco Bell.)
that's awesome. Just a thought, I wonder if users would appreciate knowing that. To me, the reason I asked is I tend to assume the worst (minimal permission, 1 person in the org, no contract, etc...). Maybe just me, but especially so when it's huge names like Amazon or Stripe
I've seen SaaS providers engage in really scummy practices here. Like comb through their access logs or user databases, find that someone signed up from an IP address belonging to a corporate ASN, and then putting that company's logo on their website as a user/customer. Technically telling the truth, but with information obtained in a really shady manner.
(I'm not claiming Retool does or does not do this.)
You need to be an insider. Like maybe your sister is married to a big tech CEO or your parents own a lot of big tech stocks. Any personal/family connection will do.
It is fairly common. They probably aren't code splitting/tree shaking their bundle very well. With that said, I assume 99.99% of their user sessions come from a desktop computer, so I doubt hardly anyone notices. They have other things to work on that are more important than optimizing their 4mb app.js script.
That's a fair concern and something we're working on improving every day! It's fairly common when you consider Retool in the context of other development platforms. VSCode is a 200MB download and is considered small in the desktop space. Even gdocs is ~30MB.
For context, these resources are cached on your machine. So you should find that the actual data transferred significantly decreases after the first visit.
I presume that figure will be including a full browser, which is obviously inapplicable if you’re opening it in an existing browser, as you can and as the cited page is.
For comparison, on Arch Linux, the code package is a 15 MiB download, 90 MiB installed, and depends on electron22 (that is, it doesn’t include a browser itself).
> and is considered small in the desktop space.
Citation needed. I don’t think anyone reasonable would consider it small, not by a long shot. Not as huge as some very large things, sure, but certainly not small.
As for Google Docs, I hope no one uses Google stuff as an example of good or justifiable largeness. I can’t speak for Google Docs at all, but Gmail is one that I was historically somewhat familiar with, and around five years ago it was generally around 10× the download size and memory usage of Fastmail’s webmail on reasonably similar functionality (with one having more in one direction and the other more in another direction), and I know a more recent change made things at least twice as bad, though they could have clawed that back by now, and Fastmail resource requirements have definitely increased somewhat since.
> It's fairly common when you consider Retool in the context of other development platforms. VSCode is a 200MB download and is considered small in the desktop space. Even gdocs is ~30MB.
Are you even reading the parent comment? He is pointing to the login page. I don't understand the comparison with VSCode.
Great to hear that! There is hope to frontend obesity crisis. What actions have you done to slip down the login page to 6MB? How large was it in the beginning? What is the bulk of that file? I mean React framework would span only 0.1% of that room.
I checked in with an engineer who’s focussing on app performance. You're right that this screen should not be 6MB. One part of the bloat is due to our webpack configuration. In lighthouse, you can see that >50% of the app.js file is unused for login. Most of this is actually used for the app after login.
We’re working on getting this down significantly but, that being said, the login page isn’t our highest priority. It’s the first thing customers see, but >99% of the time, they’ll already be logged in and navigate straight to the apps page, or a specific Retool app. For these pages, we’ve:
1. re-written our core runtime to avoid excessive message passing with iframes (up to 75% faster page load for slowest apps)
2. parallelized resource fetching with JS execution
3. cached app graph information in IndexedDB
4. tweaked webpack chunking strategy to avoid very small JS/CSS chunks
5. optimized resource fetching to avoid over fetching
Hope that’s helpful context. Definitely let me know if you have thoughts/other ideas
I imagine they have a huge task list with a lot of higher priority tasks than reducing the js bundle. Sure it'd be nice, but it's not what users most want.
Most of the use cases we are seeing LLMs unlock look trivial in isolation when looked at it from content generation point of view but become super powerful when mixed into workflows. This is evident from all the startups we are seeing lately in this space and LLM integrations into existing products like Zapier, Hubspot and now Retool AI.
Granted these are still early days for LLMs, but it looks they are primarily going to revolutionize how knowledge work is done at a scale similar to what computers have done. There are exciting projects coming out every week putting LLMs into the workflows of more and more knowledge workers. open-interpreter is the recent of the bunch.
Similar to other workflow automation projects in this space, we started building LLMStack (https://github.com/trypromptly/LLMStack) with a goal of scaling knowledge work pieplines. Exciting times ahead!
I'd love a template for:
1. Select internal document sources you want to ingest (Notion/Slab, specific Slack channels, named Google Docs in a folder)
2. Enable Q&A + augmented retrieval in response
3. Serve to end users in a Slack channel at our company
You can do this with LLMStack. We are adding the datasource handlers to pull data from Google docs etc., Slack integration is built in. More info at https://llmstack.ai/docs/apps/integrations/slack. Happy to chat if you have questions.
Agree we're working on building this exact flow into the product now. But in the meantime I have a Workflow (our version of a webhook/cron job) that does this internally that I can send it to you if you want. Just ping me at jamie AT Retool DOT com
Yes working on adding llama2 and AWS bedrock support right now for open source models. But we also want to let users bring their own models - is there a specific model you're looking for?
Who knows longer-term I'd love to help users fine-tune their own models on llama2/gpt-3.5!
Llama 2 and Bedrock is exactly what we are looking at.
I have been playing around with it for a priority enterprise use-case at my org, and this seems promising. We tried using the Retool vectors, but it really struggles to pick up the right context if we dump data without pre-processing it. For example, adding a URL without stripping header/footers/disclaimers/cross links etc. I think adding some steerability to the context selection process is going to be key.
I also have to admit that while retool is extremely feature rich - the user-experience is clunky and the learning curve is steep.
Thanks for the feedback. We're currently working on customization with the url parser and trimming headers/etc makes a lot of sense. If you're able to share more about your use case (jamie AT retool DOT com), happy to get it tuned on your dataset
(Retool PM here) Our automation product Workflows (https://retool.com/products/workflows/) now has an AI actions block so it's easy to build flows with steps like “summarize text” or “classify image”. Our approach is to make it easy to chain different steps together into flows whether those steps are code (JS/Python/SQL), API calls, database queries, or LLM prompts.
Zapier is no-code, catering more towards a citizen developer audience, where we’ve built Workflows with professional developers in mind. We typically see customers turn to Workflows when they want scalable self-hosting and more customization options (e.g. complex branching/looping, source control, custom error handling) than what Zapier is offering.
Not directly imo. Zapier focus is still on workflow automation and retool is on internal tools. This directly competes with a bunch of startups that recently launched including an open-source project we put out a few weeks back (https://github.com/trypromptly/LLMStack)
Looks awesome, congrats! Do you provide any integrations for identifying hallucinated outputs and/or showing references with the output (like Bing does)? Such a feature is critical for AI outputs, so it might be nice if it's supported out of the box.
Thanks! Yes agree this is very important. We do show references from Retool Vectors out of the box, so you can see what embeddings were used to generate a response. If it'd be helpful I can walk you through a few internal Q&A bot examples which display references - jamie AT retool DOT com
Excited to already see some great open source projects in this space (Langchain, Llamaindex, Axilla, etc) many of which we're exploring using for this project.
Just to be clear Retool Vectors is completely free (we're even taking on the LLM costs). We put the embeddings in a free postgres DB (and soon other open source vector DB providers) and give you the connection string so you can take it anywhere
Today we use a cosine similarity search but we're planning to integrate with more vector databases and allow you to customize the search logic soon. Is there a particular query method you're looking for?
I then tried building a custom AI chatbot for our support team, powered by a vector database (with previous answers, our docs, and community forum data). In my prototypes, the successful response rate was much higher (around 5 - 7x higher, ~30%). But it was surprisingly hard to build this bot: I needed to stitch together vector databases, custom integrations with Intercom, cronjobs to sync data, etc.
That’s why we built Retool Vectors. The idea is that we want to build a vector database that has full ETL from whatever inputs you have (e.g. Intercom, a postgres database, Salesforce, a community forum, a website, etc.). It’s always kept up to date. (We’re still working on some of these features, but I decided to try and launch this week because I just wanted to get feedback from HN, haha.) I think the industry has now settled on vector databases as the best way to provide context to LLMs. I hope that Retool Vectors can be a much easier way of getting data into it.
If you have any feedback please let me know!