| 7 Apr 2021 |
 fulmicoton (Paul Masurel) | bytes or string in base64 | 04:00:34 |
 willeml | Yeah, ended up using bytes, working great btw, I was able to try a few queries, and am very happy | 04:01:07 |
| fulmicoton (Paul Masurel) | I am glad to hear that!!! | 09:33:57 |
| 9 Apr 2021 |
| willeml set a profile picture. | 17:23:50 |
| 14 Apr 2021 |
|  ChillFish8 (Harrison Burt) joined the room. | 14:18:15 |
| ChillFish8 (Harrison Burt) | Hi all 👋 I might me missing omsething but im trying to work out how to maintain a persistent document store? | 14:18:17 |
| willeml | Do you mean persistent as in the data is kept even after restarting your program? | 15:15:30 |
| willeml | Or just persistent across threads | 15:15:33 |
| ChillFish8 (Harrison Burt) |
Do you mean persistent as in the data is kept even after restarting your program?
This^ | 15:26:18 |
| ChillFish8 (Harrison Burt) | Thought i'd check if tantivy has support for it before making a system | 15:26:39 |
 maufl | You can use `Index::open_or_create` to open an existing index in a directory. As long as the content of this directory is not deleted between restarts of your program, the index and the stored parts of your documents should persist. Does this answer the question? | 16:32:39 |
| ChillFish8 (Harrison Burt) | yes thanks alot 👌 | 17:38:41 |
| fulmicoton (Paul Masurel) | This
E | 22:03:07 |
| fulmicoton (Paul Masurel) | This is correct. Make sure you so no | 22:03:22 |
| fulmicoton (Paul Masurel) | Arg | 22:03:32 |
| fulmicoton (Paul Masurel) | Make sure you do not forget to commit() | 22:03:46 |
| fulmicoton (Paul Masurel) | Tantivy guarantees that your index is persisted and does not get corrupted on failure like: | 22:04:36 |
| fulmicoton (Paul Masurel) | Your process or the computer crashed was killed at any timing (during commit or before commit) | 22:05:13 |
| willeml | That reminds me, is there a better way of doing ```rust
let dir_string = format!(
"{}/{:x}/{:x}/index",
crate::hub::HUB_DATA_FOLDER,
hub_id.as_u128(),
channel_id.as_u128()
);
let dir_path = std::path::Path::new(&dir_string);
if !dir_path.is_dir() {
std::fs::create_dir_all(dir_path)?;
}
let dir = MmapDirectory::open(dir_path).map_err(|| DataError::Directory)?;
let index = Index::open_or_create(dir, self.schema.clone())
.map_err(|| IndexError::OpenCreateIndex)?;``` | 22:05:50 |
| willeml | (MmapDirectory part) | 22:06:23 |
| fulmicoton (Paul Masurel) | It looks ok to me | 22:07:18 |
| willeml | What is the MmapDirectory (why is it not just directory) | 22:07:58 |
| willeml | I am asking if it’s fine because I just used the only impl of directory I could find | 22:08:27 |
| fulmicoton (Paul Masurel) | Sure. Tantivy does all of its io through an abstraction called directory | 22:37:25 |
| fulmicoton (Paul Masurel) | 90% of the time, you want to use the mmapdirectory which relies on memory mapping to do the io on your hard disk. | 22:38:21 |
| fulmicoton (Paul Masurel) | For unit test you probably do not want to create a bazillion temp directories. The RAMDirectory is preferable. It is not persisting your data of course. | 22:39:37 |
| fulmicoton (Paul Masurel) | That is all that is included in tantivy , but it is possible to extend tantivy with your own implementation. | 22:40:23 |
| fulmicoton (Paul Masurel) | For instance Quickwit (my startup) has 4 implementation of directory to make it possible to search from an amazon S3 bucket. | 22:41:32 |
| willeml | Awesome, thanks for the clarification! | 22:55:31 |
| 15 Apr 2021 |
|  lpouget joined the room. | 07:05:43 |
