!JxzWpvsvfYuMHyUJiL:matrix.org

rust-ipfs

367 Members
Implementing ipfs in rust44 Servers

Load older messages


SenderMessageTime
16 Mar 2023
@rklaehn:matrix.orgrklaehn@mindtree your best bet is https://github.com/n0-computer/beetle . `iroh` is no longer aiming for unixfs compatibility.04:24:05
@rklaehn:matrix.orgrklaehnIpfs-embed itself is solid and is being used in production for years. But it has minimal support for unixfs (just reading).04:25:43
@rklaehn:matrix.orgrklaehn(Beetle is the former go-ipfs compatible iroh)04:26:51
@paulali:matrix.orgpaulali mindtree: I think it depends on your usecase, if the current state of Iroh meets it that is okay. The rust implementation is a bit behind compared to where kubo support currently is. You could check out https://docs.rs/ipfs-unixfs/latest/ipfs_unixfs/ to see if you could use it to fit your needa 04:30:13
@paulali:matrix.orgpaulali* mindtree: I think it depends on your usecase, if the current state of Iroh meets it that is okay. The rust implementation is a bit behind compared to where kubo support currently is. You could check out https://docs.rs/ipfs-unixfs/latest/ipfs_unixfs/ to see if you could use it to fit your needs04:30:25
@rklaehn:matrix.orgrklaehnWe (n0) got a decent gateway with unixfs read and write support including HAMT. I think the unixfs part is pretty complete. Where we had issues is 100% interop with go-ipfs regarding bitswap, due to some rust-libp2p issues beyond our control.04:35:16
@mindtree:matrix.orgmindtree

Thanks both for the SITREP!

To give a little more context on my application - I'm working on a source code package manager similar to cargo, but that supports IPFS as one of the source types. Eventually, the idea is that our registry will be based on IPFS, i.e. a minimal map from name+version to CIDs (the registry itself on some blockchain).

UnixFS struck me as a good fit, as I could just "insert" the package's directory when publishing, and "get" it when fetching source.

I was hoping to use something compatible with go-ipfs, so that users could use existing tooling to easily mirror/pin registry packages that they care about, but it sounds like we might not have an implementation capable of both insertion and retrieval in a go-ipfs-compatible manner 🤔

For now I might even just try compressing the source directory into a slice of bytes and then insert and get that raw slice using ipfs-embed. Certainly not merkle-daggy or very granular, but these packages will mostly just be src files so probably not a big issue anyway.

Does this sound crazy? I'll likely revisit UnixFS again in the future, but at least this would get us started

04:44:00
@mindtree:matrix.orgmindtree

rklaehn: I just watched your blake3 and bao overview earlier - thanks for sharing, was cool to get an overview of the guts :)

Where we had issues is 100% interop with go-ipfs regarding bitswap, due to some rust-libp2p issues beyond our control.

Is there somewhere I can read about what these issues were?

It sounds like you guys are currently focused on maximal performance for p2p content-addressed storage in general. Is the idea that once you've got this lower-level implemented, you'll look into ipfs-compatibility on top? Or do you see iroh as a whole new alternative to IPFS?

04:48:36
@rklaehn:matrix.orgrklaehnWell, we see it as fulfilling the original promise of ipfs - a global content-addressed file system.04:50:25
@rklaehn:matrix.orgrklaehnIf someone wants to put unixfs on top, no problem. But go-ipfs compat is not planned.04:51:49
@rklaehn:matrix.orgrklaehnWe spent a lot of time writing a high performance bitswap impl, only to notice that it can not be really high performance and follow the go-ipfs approach. We also had issues with go-ipfs and rust-libp2p interop at the stream level. At some point it was just too much...04:55:35
@rklaehn:matrix.orgrklaehnDon't know the exact details, since I was not doing that part. But for ipfs-embed by David and me we also found go-ipfs compat hard and finicky.04:57:05
@rklaehn:matrix.orgrklaehnGlad you liked the video. It was the first take where the recording worked, so it was not perfect. But I don't have time to redo it - have to write BAO stuff...04:59:30
@mindtree:matrix.orgmindtree
In reply to @rklaehn:matrix.org
Well, we see it as fulfilling the original promise of ipfs - a global content-addressed file system.

Yeah I can see where you're coming from for sure, and I'm excited to see what you folks come up with!

The main concern I have for iroh is that I worry folks already associate "IPFS" with not just the idea, but the go implementation. There's already a lot of tooling around the go implementation like the CLI, the browser extension, etc, and when a lot of folks think IPFS I'm guessing they just think of those tools.

If iroh is to dismiss compatibility with all of this tooling, that seems like a whole other level of ambition as I guess it puts you in a position where you're now competing not only against the go implementation, but against the whole ecosystem's tooling, its marketing, and the existing mindshare where "go-ipfs = IPFS"?

If that's the plan, then I have a lot of respect for iroh in going for it! But I do worry that people will get confused when they're looking for an IPFS implementation, come across iroh which describes itself as an IPFS implementation, they try using it, and then wonder why it doesn't work with "all the other IPFS stuff".

It would be cool to see a compatibility table between kubo and iroh's plans. E.g. is it just that the two will have different networks because of their DHT implementation differences? Or is it the case that even the CIDs for the same raw data are different between the two?

05:12:01
@rklaehn:matrix.orgrklaehnWe are figuring it all out right now. Expect more clarity by the time of ipfs thing in April.05:13:33
@rklaehn:matrix.orgrklaehnOur Cids are blake3 Cids, and our blocks can be of arbitrary size. Other than that there is nothing unusual about them.05:14:39
17 Mar 2023
@siman:matrix.org@siman:matrix.org left the room.00:56:42
@david:web3.foundationDavid | W3F - OOO May 13 changed their display name from David to David - OOO (sick).06:10:12
18 Mar 2023
@torir:matrix.orgTorir joined the room.22:27:55
@torir:matrix.orgTorir changed their display name from torir to Torir.22:47:13
20 Mar 2023
@musicmatze:beyermatthi.as@musicmatze:beyermatthi.as left the room.15:47:21
22 Mar 2023
@david:web3.foundationDavid | W3F - OOO May 13 changed their display name from David - OOO (sick) to David | W3F.08:07:13
23 Mar 2023
@chasev:matrix.org@chasev:matrix.org joined the room.00:19:20
@opengalgogames:matrix.orgGames joined the room.14:29:29
28 Mar 2023
@hamnox:matrix.org@hamnox:matrix.org left the room.23:32:44
5 Apr 2023
@ivii:matrix.orgivii joined the room.15:37:04
6 Apr 2023
@sergd85:matrix.orgsergd85 joined the room.03:34:43
7 Apr 2023
@david:web3.foundationDavid | W3F - OOO May 13 changed their display name from David | W3F to David | W3F - OOO Apr 13.07:53:12
10 Apr 2023
@cryptoquick:matrix.orgHunter Beast rklaehn: I found a version of Bao once that implemented batch slice verification, have you seen it? I forget it's name, but I figured you might know. Also, do you know if the existing crate is still being maintained? 21:42:20
11 Apr 2023
@rklaehn:matrix.orgrklaehnWdym batch validation? This would be quite easy to implement using my crate bao-tree, but one of the main attractions of BAO is streaming validation as data comes in 11:27:08

Show newer messages


Back to Room ListRoom Version: 5