Thread

I keep forgetting to use it, but I'm very tempted to port my current relay project over

You won't regret it.

Do you have thoughts on migrating from go-nostr to nostrlib?

In any case, if you want it this should be a good start:

GitHub

migrate from go-nostr to nostrlib by staab · Pull Request #4 · pippellia-btc/rely

I've started using go-nostr as it was the main (and only) nostr library for go, also because I used khatru before building rely. I've contributed 3 PRs to it, each time after seeing my projects panic in production. The first time, because the event CheckID method panic when the event had a shorter ID. The second time because it tried to access a tag without checking it was there. And the third, because I've discovered a data race that originated from completely misusing goroutines.

GitHub

Fix race in `ConnectWithTLS` where `relay.Connection` can be set to `nil` concurrently with writes, causing panics. by pippellia-btc · Pull Request #183 · nbd-wtf/go-nostr

I am running go-nostr v0.49.7 and recently got multiple times this panic: May 04 22:47:20 vertex crawler[13698]: panic: runtime error: invalid memo...

The first two were silly mistakes, but the third really highlights a pattern I found in the whole library: The abuse and misuse of goroutines, the overcomplicated and convoluted patterns to provide synchronization. I've looked at the new nostrlib, admittedly not much, but I've sees the same bad patterns over and over. So, my current thinking is that, if I'll migrate, I'll probably migrate to a simpler library that only provides the data structures for nostr, which is the only thing rely uses. Also because the JSON encoding / decoding can be made much more memory efficient, which would benefit relays a lot. So yeah, long rant 😝

First of all, thank you for this PR. As I've explained in my other message, I'm not sure if I'll accept it. Questions for you. How bad would it be for you to migrate your own DB to using a new library? Do you use many of the feature of go-nostr / nostrlib provides other that structure methods (Sign, CheckID...)? If so, which ones?

No, that makes sense. fiatjaf's libraries tend to have some super important primitives in them (like the event and filter structs) alongside very opinionated stateful stuff like khatru and nostr-tools' relay connection logic. It would be nice to have a nostr basics library with a much smaller scope that everyone could agree on so you don't have to write adapters between other libraries.

No, it's mostly just the core structures that I use. I could migrate to go-nostr, but nostrlib is genuinely better in this area. I attempted to switch to rely this morning and decided against it for now due to the lack of negentropy and blossom support. I get not wanting to include that stuff, but examples or external libraries would make the process easier (I do think negentropy should be table stakes for relays nowadays).

Yeah It would be nice. Not to mention the dependencies, which is something I try to be aware. go-nostr and nostrlib have 30+ dependencies, 2 different libs for websocket, and 3 for json! Like wtf

Thanks for the feedback. To me, blossom is an orthogonal thing, which can be added in this way: - create an http server (there are many many frameworks in go, even the standard library is fine). - create a relay - use the relay handlers in the http server (relay.ServeWS, relay.ServeNip11) - add the blossom handlers to the server and voilà, server is now relay + blossom. For negentropy, that is another story since it used the websocket protocol. It would be a nice addition to rely, so I'll add it to the roadmap. I am just conscious about complexity in general, since I want to maintain rely long term, not leave it rotting.

How about a new library that has only data structures, no dependencies, minimal utilities like a relay pool, Query, Subscribe, and very performant JSON marshalling / unmarshalling? Man I am so tempted to build this

I would entirely skip the networking code, but yes

Absolutely, I think that breakdown is the right call. A blossom library could be built separately

my relay #orly has a blossom server built into it now. the library has the fastest event codecs for both the database binary version and the json, i just finished adding a feature where it can now load the libsecp256k1 library as a dynamic linkable without the cgo complications (you just need a version of the library with schnorrsig extension on). it doesn't have negentropy but i have built a sync protocol that lets you build a cluster that stays in sync, and using the new policy engine you can specialise nodes to only replicate some specific criteria of events. i probably should put negentropy in there. easy enough to copy it out of khatru.

oh yeah, i forgot, the orly repo libraries also have an NWC client implementation, and it's used in a subscription whitelist system.

users pay for subscription by zapping the relay's npub btw.

man, how about we collaborate on a simple library with only data structures, JSON encoding and cryptography? A standalone library would be easier for others to use.

my work is CC0 :) feel free to copy the algorithms and tidy up the API, i'm familiar with it in its current state but maybe it can be polished more. one of the things that will be unfamiliar is that it uses raw binary formats for the fields that are hex in the events. this means their comparisons happen twice as fast. but it throws up some gotchas for young players and plenty of times the LLM mangles things and is comparing the raw decoded binary with the not-decoded hex. also, the JSON codecs for filter and event take a lot of strict checking shortcuts because basically nobody uses json codecs that make these errors although they always have these linter checks through the encoder. and i wrote them using labels and goto btw :) much faster and more memory efficient for simple state machines. the other thing is the new "purego" secp256k1 library i have put together - it dynamically loads the binary library if it can find it and uses the much faster C code from bitcoin core. it complicates deployment in that to make use of it you have to put the library in there, and ideally you enable all modules, but it must have schnorrsig enabled. if it doesn't find the library it should fall back to my new pure go library that does all the key nostr crypto operations faster than btcec. i'll just stick to using next.orly.dev directly for other things but if you make the same performance available with a more polished UX that would be fantastic.

Okay thanks, at some point in the future I'll give it a go.

yeah, btw, i intend to have it so you can use the code and run the relay on all platforms, linux, windows, mac, android and ios. on ios it needs to fall back to the swift implementation and probably the kotlin one on android but the dynamic loader works for all desktop platforms. i want to make mobile apps. i'm also concurrently working on a fresh new Vulkan based immediate mode GUI library with a neat and concise fluent chained methods API, i expect in a few months that will have reached the point where i can actually build a small simple app with it and it's pure go, no CGO, and faster than fear

(and will let you have one codebase that includes the web app version)

That's cool, but I am mostly interested in the low level foundational nostr library as I said

yeah it's only for the cryptography... signatures, hash functions. a lot of stuff SIMD or fast pure C it's worth having them available if they are found i think. when i used btcec to derive pubkeys to get this key i have now, it took over 3 days. now i can generate a -mleku key in about 5 minutes max, by using the libsecp256k1. so, anyhow, as far as that goes, i've got the crypto sorted out to be as fast as possible because verifying events especially is a huge part of the time spent processing

Thanks for the explanation, I was thinking "why not khatru" during all the video. It's very useful to know why you decided to build a new project from the grounds up. I recall @ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ also had sync issues with khatru, and he also builds a relay, called 'realy'. Now really, we have rely, realy as relays. I guess it's not easy to google them if something goes wrong... :D

but better

yeah, that's an interesting idea. But what do you mean by personal? A relay that accepts only your notes? Maybe your notes and the ones of your follows? A relay that proactively fetches your follows posts? A relay that does DMs as well?

i've used haven from bitvora in the past, but that seems overkill. but tbh i wanted to get into nostr dev since months, but i just didn't take the time to do so. that means i don't know exactly what i would like in a personal relay, i just didn't think about it that much. i am big into self-hosting and would like something at relay.[mydomain] which i can point clients to also this was just an idea that came to my mind when i was watching your video introducing rely v1. i would probably watch a stream/video of you developing something with rely. but in no way i want you to do something, just because i asked for it

no worries sir, I just wanted to understand what you were looking for. I think rely is a great great start if you want to do nostr development. There are many examples in the /examples directory, all very simple to follow.

#orly has a "follows ACL" and a sync spider that stays very up to date, it whitelists all your follows, and automatically searches the events for relays and searches for events from them that it doesn't have, and syncs. within about 5 seconds after starting it up, it updates everything it can find. it was specifically designed to be a personal relay, my first purpose with orly has been that. it has many more production-oriented features now, but the personal relay use case is my highest priority.

will take a look at orly

https://next.orly.dev/ btw :)

also, btw, the sync is so fast that if it is published to one of the relays its subscribing for events from your follows or yourself (you need to follow yourself, i think, this should just be default though) that the subscription to that other relay is forwarded immediately and ORLY will respond that it already has the event. happens under 200ms