!wZBXZwtTJAgBQYuVGZ:matrix.org

#tantivy:matrix.org

80 Members
4 Servers

Load older messages


SenderMessageTime
7 Apr 2021
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) bytes or string in base64 04:00:34
@willeml:matrix.orgwillemlYeah, ended up using bytes, working great btw, I was able to try a few queries, and am very happy04:01:07
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) I am glad to hear that!!! 09:33:57
9 Apr 2021
@willeml:matrix.orgwilleml set a profile picture.17:23:50
14 Apr 2021
@chillfish8-5f7b8e38d73408ce4ff0c155:gitter.imChillFish8 (Harrison Burt) joined the room.14:18:15
@chillfish8-5f7b8e38d73408ce4ff0c155:gitter.imChillFish8 (Harrison Burt) Hi all πŸ‘‹ I might me missing omsething but im trying to work out how to maintain a persistent document store? 14:18:17
@willeml:matrix.orgwillemlDo you mean persistent as in the data is kept even after restarting your program?15:15:30
@willeml:matrix.orgwillemlOr just persistent across threads15:15:33
@chillfish8-5f7b8e38d73408ce4ff0c155:gitter.imChillFish8 (Harrison Burt)

Do you mean persistent as in the data is kept even after restarting your program?

This^

15:26:18
@chillfish8-5f7b8e38d73408ce4ff0c155:gitter.imChillFish8 (Harrison Burt) Thought i'd check if tantivy has support for it before making a system 15:26:39
@maufl:maufl.demauflYou can use `Index::open_or_create` to open an existing index in a directory. As long as the content of this directory is not deleted between restarts of your program, the index and the stored parts of your documents should persist. Does this answer the question?16:32:39
@chillfish8-5f7b8e38d73408ce4ff0c155:gitter.imChillFish8 (Harrison Burt) yes thanks alot πŸ‘Œ 17:38:41
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) This
E
22:03:07
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) This is correct. Make sure you so no 22:03:22
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) Arg 22:03:32
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) Make sure you do not forget to commit() 22:03:46
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) Tantivy guarantees that your index is persisted and does not get corrupted on failure like: 22:04:36
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) Your process or the computer crashed was killed at any timing (during commit or before commit) 22:05:13
@willeml:matrix.orgwilleml

That reminds me, is there a better way of doing ```rust

let dir_string = format!(
"{}/{:x}/{:x}/index",
crate::hub::HUB_DATA_FOLDER,
hub_id.as_u128(),
channel_id.as_u128()
);
let dir_path = std::path::Path::new(&dir_string);
if !dir_path.is_dir() {
std::fs::create_dir_all(dir_path)?;
}
let dir = MmapDirectory::open(dir_path).map_err(|| DataError::Directory)?;
let index = Index::open_or_create(dir, self.schema.clone())
.map_err(|
| IndexError::OpenCreateIndex)?;```

22:05:50
@willeml:matrix.orgwilleml(MmapDirectory part)22:06:23
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) It looks ok to me 22:07:18
@willeml:matrix.orgwillemlWhat is the MmapDirectory (why is it not just directory)22:07:58
@willeml:matrix.orgwillemlI am asking if it’s fine because I just used the only impl of directory I could find22:08:27
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) Sure. Tantivy does all of its io through an abstraction called directory 22:37:25
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) 90% of the time, you want to use the mmapdirectory which relies on memory mapping to do the io on your hard disk. 22:38:21
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) For unit test you probably do not want to create a bazillion temp directories. The RAMDirectory is preferable. It is not persisting your data of course. 22:39:37
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) That is all that is included in tantivy , but it is possible to extend tantivy with your own implementation. 22:40:23
@fulmicoton-529c6b27ed5ab0b3bf04d55b:gitter.imfulmicoton (Paul Masurel) For instance Quickwit (my startup) has 4 implementation of directory to make it possible to search from an amazon S3 bucket. 22:41:32
@willeml:matrix.orgwillemlAwesome, thanks for the clarification!22:55:31
15 Apr 2021
@lpouget:matrix.orglpouget joined the room.07:05:43

There are no newer messages yet.


Back to Room List