!wZBXZwtTJAgBQYuVGZ:matrix.org

#tantivy:matrix.org

60 Members
1 Servers

Load older messages


SenderMessageTime
9 Jan 2021
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter)if you tend to commit independently from your change (ie: on a different thread, taking a lock on the indexwriter)00:22:40
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter)you can send both operation at the same time to ensure that your upsert looks like a transcation00:23:28
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter)and at no time someone sees your index in a state where you have two docs with the same id, or 0 doc with the same id00:23:52
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter) if you do delete , add , commit_ in the same thread, you will get that property too of course. 00:24:38
@Follpvosten:matrix.orgFollpvostenSo my only operation is delete, then re-add, and currently I'm acquiring a writer for each of these00:26:44
@Follpvosten:matrix.orgFollpvostenAh, and so using IndexWriter::run does have a benefit, interesting 00:27:06
@Follpvosten:matrix.orgFollpvostenI kinda assumed it didn't really matter since it's all sent on commit()00:27:36
@Follpvosten:matrix.orgFollpvostenWhen it comes to threads, I think there's an await-point between the change being saved in the database and updating the search index, so it might thread-hop there in theory00:29:22
11 Jan 2021
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter)

So my only operation is delete, then re-add, and currently I'm acquiring a writer for each of these

If it is a service, please make a writer once and for all and reuse it.

10:24:36
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter)it will require locking10:24:39
@Follpvosten:matrix.orgFollpvostenWhat defines a "service"? My wrapper around the index, reader etc. will regularily jump between threads, so it needs to be Send + Sync15:04:35
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter) I mean an ever running program as opposed as a CLI 15:30:22
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter) Your use an arc< rwlock> 15:31:02
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter)You could use*15:31:15
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter)The best option requires a lot more plumbing.15:31:47
@Follpvosten:matrix.orgFollpvostenYeah then it is a service (which is why an in-memory index works well for me)17:33:30
@Follpvosten:matrix.orgFollpvostenI think I'll want to benchmark how much I'd gain from an Arc<Mutex<>> there 17:34:11
@Follpvosten:matrix.orgFollpvosten(RwLock doesn't really make sense as it's read optimized and the writer will only ever be used as mut)17:34:45
@Follpvosten:matrix.orgFollpvostenFact is, I don't expect many writes at all, at least at first17:35:15
@Follpvosten:matrix.orgFollpvostenSo keeping the writer around might be an optimization for later17:35:39
12 Jan 2021
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter) IndexWriter::add_document is not &mut 02:06:04
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter) I strongly recommend using an Arc<Mutex<...>> and keeping the same IndexWriter 02:08:04
@gitter_fulmicoton:matrix.orgPaul Masurel (Gitter)this is not just about performance02:08:13
@Follpvosten:matrix.orgFollpvosten
In reply to @gitter_fulmicoton:matrix.org
IndexWriter::add_document is not &mut
Well I need to commit immediately, always
03:18:53
@Follpvosten:matrix.orgFollpvosten
In reply to @gitter_fulmicoton:matrix.org
I strongly recommend using an Arc<Mutex<...>> and keeping the same IndexWriter
But alright, I'm not against it
03:19:09
16 Jan 2021
@rajyogi:matrix.orgrajyogi joined the room.02:26:24
17 Jan 2021
@gitter_matthew:matrix.orgMatthew (Gitter) joined the room.12:17:07
* @gitter_matthew:matrix.orgMatthew (Gitter) waves 12:17:08
@gitter_matthew:matrix.orgMatthew (Gitter) fulmicoton: i just saw https://twitter.com/fulmicoton/status/1350776034094813184?s=20 12:17:22
@gitter_matthew:matrix.orgMatthew (Gitter)i think the problem running tantivy in a browser via wasm is the threading; do you think that is a tractable problem?12:18:17

There are no newer messages yet.


Back to Room List