Which model for a RAG chatbot needing to get information from a database?

Beginners
1

Hi everyone,

I am rather new here, so please excuse basic questions or requests for clarification.

We are a small foundation and we operate an observatory providing information on political parties and their funding. We were recently granted a VPS as an in-kind donation and we are trying to use it to run a chatbot that would help answer users’ questions.

The idea is that, since the data can be complex, users could directly quiz the bot to get specific answers and, ideally, to draw charts based on this data.

For now we have AnythingLLM set up on the VPS and we are starting to play with it. Now we are trying to figure out what model can use (with an inference point, right?) to answer the queries.

Any comments and suggestions are welcome!

1 Like
2

When the primary use case involves database lookups, correctly integrating the LLM with its surrounding components often becomes more critical than the LLM’s own granular performance.

Regarding LLM selection itself, models that are relatively new (post-2025), have 7B parameters or more (if possible), and are Instruct models (pre-fine-tuned as chatbots) generally pose few issues.
https://huggingface.co/models?num_parameters=min:3B&apps=vllm,ollama&sort=trending

3

This happens to be the latest model released just a few days ago, and since it seems to excel at tool calling, it might be usable even for use cases like this one. It’s a super-compact 270M LLM, so I think it should be over 20 times faster than a 7B LLM…

Of course, its actual capabilities as a chatbot probably aren’t very high.:sweat_smile: