Thread - Hyper

Introducing a little weekend project: Cuaklay. This is an event store for relays written in Go using DuckDB. You can use it as storage for your Khatru or Relay implementations. It supports NIP-50 search out of the box, and you can also use it standalone for pure analytical purposes. It's quite fast.

GitHub

GitHub - gzuuus/cuaklay

Contribute to gzuuus/cuaklay development by creating an account on GitHub.

The idea behind this project is to evolve and leverage DuckDB's analytical capabilities even more. The goal is to allow you to create algo feeds based on your own criteria, crunch stats on an event dataset, or perform any other analytics on Nostr events. Currently, it only has search capabilities, which, when combined with other fields from Nostr filters, make it quite powerful. Future improvements will bring more query capabilities, probably opt-in BM25 FTS support, and more. To get started, just go to the repository and read the instructions on how to use it to integrate it into your relay implementation, use it standalone, or even spin up a relay that's already built in as a separate module, all the info and instructions are in the repo. Hope you find this interesting, this is just the beginning!

Replies (2)

hzrd149 🛡️ 2 days ago

What type of indexing are you using for the NIP50 search? Is it similar to SQLghts FTS5?

Gzuuus 🛡️ 2 days ago

↩ replying to hzrd149

No index is currently used, the search relies on DuckDB's pattern matching capabilities with ILIKE. Therefore, no index is needed, and the data is always fresh. DuckDB's columnar storage makes this approach performant. However, it is true that performance will degrade gracefully as the dataset grows. Through testing, I found that you can handle quite a large dataset, over +100k and still maintain good performance. I am considering introducing BM25 fts. However, this approach relies on indexes that would need to be rebuilt each time to ensure it is fresh. My initial idea for implementation is to make this optional through configuration and integrate it into the current search function. Something like: If the BM25 index exists, it will be used, otherwise, it will fall back to pattern matching.