| 29 Feb 2020 |
 @yegortimoshenko:matrix.org | https://github.com/tweag/funflow
Funflow's caching is based around the content store. This stores all artifacts according to the hash of the inputs and computations which produced them. Funflow uses this to know when exactly a computation must be rerun and when previous results can be re-used.
| 15:31:35 |
 @arc:arcn.mx | mew o: | 15:38:43 |
| @yegortimoshenko:matrix.org | (you all have probably already seen that... metadata is stored in sqlite, blobs are plain fs files, they model deps as lists: https://git.io/JvgVo) | 15:43:40 |
| @yegortimoshenko:matrix.org | * (you all have probably already seen that >_> metadata is stored in sqlite, blobs are plain fs files, they model deps as lists: https://git.io/JvgVo) | 15:44:03 |
| @yegortimoshenko:matrix.org | * (you all have probably already seen that >_> metadata is stored in sqlite, blobs are plain fs files, they model deps as lists) | 15:45:47 |
| @yegortimoshenko:matrix.org | * (you all have probably already seen that >_> metadata is stored in sqlite, blobs are plain fs files, deps are modeled as lists) | 15:46:50 |
 emily | I don't think I actually had seen this | 15:59:48 |
| emily | (though I have seen similar stuff) | 15:59:55 |
| emily | reminds me of https://github.com/salsa-rs/salsa | 16:00:31 |
| emily | though more elaborate (in a good way!) | 16:00:36 |
| emily | one of these days I'm going to have to get over my aversion for arrow notation I guess | 16:02:34 |
| 1 Mar 2020 |
|  @freenode_energizer`:matrix.org joined the room. | 21:17:38 |
| @freenode_energizer`:matrix.org | hello. i saw mew docs emphasize object capabilities. i am curious if something like https://github.com/NixOS/nix/issues/8#issuecomment-145911503 is in scope for mew | 21:25:20 |
| emily | I've definitely thought about whether it would be practical to make the store not world-readable, yeah | 21:25:57 |
| emily | I'm not sure it's ultimately the correct approach to take for secrets but it's something I'm interested in exploring | 21:26:10 |
| emily | the plan is to not use the filesystem as backing storage so there'll need to be some amount of filesystem namespacing / FUSE / ... going on to begin with | 21:26:42 |
| emily | (there are some uses outside of general secrets -- e.g. if part of your code is trade secret and you want to restrict access while still allowing general builds, google actually has this -- though admittedly I don't know of any examples I'd consider particularly compelling) | 21:27:58 |
| @freenode_energizer`:matrix.org | meaning someone else is allowed to build my code but doesn't get access to its source? | 21:34:30 |
| @freenode_energizer`:matrix.org | or just that someone else is not allowed to see my stuff even though we share the same bash | 21:38:25 |
| @freenode_energizer`:matrix.org | (and libc, etc) | 21:38:41 |
| emily | yeah, along the lines of the former, secret libraries that get linked to. but it's kind of marginal and of course you can't defend against decompilation, hence why I said I don't feel like it's a particularly compelling usecase | 21:40:19 |
| @freenode_energizer`:matrix.org | mhmm | 21:41:32 |
| emily | (also, hi! I'm curious how you found the repo since I haven't really advertised it around anywhere ^^ ) | 21:44:28 |
| @freenode_energizer`:matrix.org | you helped me a few days ago in #nixos and i took a look at your github | 21:45:37 |
| @freenode_energizer`:matrix.org | and got interested :-) | 21:45:52 |
| emily | hopefully I'll find the time/motivation to make it more code and less ideas soon :) | 21:46:57 |
| @freenode_energizer`:matrix.org | i think nix could be a good data science tool (like http://dvc.org/) if it were easy to keep 100GB files in the store. in that domain, data privacy rules may disallow publication of raw data sets, but permit publication of summary tables. under "differential privacy" the threat of deanonymization (analogous to decompilation) is mitigated | 21:48:52 |
| emily | that's an interesting use case! I definitely think that systems like Nix are really important for ensuring long-term reproducibility of scientific results. storing objects of hundreds of gigabytes in the store would be a pretty big technical challenge in a few ways, though I guess you'd probably be processing them on a big enough machine that you can bear hashing them | 21:52:40 |
| emily | the store changes I have planned for mew are more along the lines of making it viable to have more and smaller objects in the store (because of caching a lot more during evaluation) -- so much "DB-y" -- rather than supporting larger files, which would be closer to the FS end of things | 21:54:11 |
| emily | (but outsourcing large blobs to the FS would be easy too, just add more complexity than I wanted to deal with for an initial draft) | 21:54:28 |
