shorten batch to 5 indices#168
shorten batch to 5 indices#168adriano-di-giovanni wants to merge 3 commits intolevelgraph:masterfrom
Conversation
|
I'd like @BigBlueHat and @jmatsushita to review this. If you folks are happy, we can merge and release. I think this is a semver-major change. |
|
Hi there, The reasoning seems spot on and I've played with changing the index resolution too to introduce a sequence number and when I did I was fairly impressed by the tests. I agree about the Also agree this should be a major semver change. Thanks for contributing! Jun |
lib/utilities.js
Outdated
| pso: ['predicate', 'subject', 'object'], | ||
| ops: ['object', 'predicate', 'subject'], | ||
| osp: ['object', 'subject', 'predicate'] | ||
| // osp: ['object', 'subject', 'predicate'] |
|
If you would like to do the renaming and document the feature, go ahead! Would you like to help maintaining this module? this is Open Open Source! |
|
I'll leave this open for a couple of days to leave everyone the chance to say something, then I'll merge and release. Is there anything else we want to push out as semver-major? |
|
@mcollina removed the line and changed I'd love to maintain it but I think it'd very hard for me to, at the moment. @mcollina I'd like to have a chat with you about your talk for jsDay. Do you think it's possible? |
|
@adriano-di-giovanni ping me via email. |
@jmatsushita what do you mean by changing the index resolution and introduce a sequence number, please? |
|
Let's wait to merge. Maybe we can just use In the meanwhile, let me recap to have you onboard :) You can search the hexastore by @giggioz and me ended up using only three indices to perform all of the above searches. The indices are
You can try yourself. How's it possible? |
I meant changing the
I was experimenting with the idea to use levelgraph as an event store. This would require generating a unique index (sequential is nice and simple) for each submission. With these semantics submitting the same exact triple twice would create 2 different entries in levelgraph and it would be possible to read the store "from the beginning" to reconstruct some state. This makes even more sense with a quad where the metadata/graph attached to the triple might be different. Basically something like what datomic does. |
|
@adriano-di-giovanni there is a catch. The query-planner can optimize some queries by using the additional indexes and leveraging the fact that the indexes are ordered. I'm not sure if this change affects that. |
|
@mcollina one of the authors replied. They use 6 indices because of an implementation detail (they use hashes to store them). Otherwise, 3 indices suffice (SPO, POS, OSP). Please, let me know if you want me to update the pull request. |
|
@adriano-di-giovanni please do. I fear breakages, but let's try :) |
|
I tried. There's a failing test. I need time to understand why. In the meanwhile you can try yourself changing |
@mcollina, this is my pull request about #167.
As you can see, I removed last item from the
defsarray. Doing so, I shortened the batch to 5. I updated tests accordingly and everything seems to work.I'd like to revisit the optimization with you: if I understand your code, you cycle the
defsarray to find just ONE index to query. If I'm correct, when input pattern is{ subject: 'subject', object: 'object' }there's no chance forOSPindex to be elected (the last item, the one I removed); you always electSOP. That's why I think that 5 indices suffice.Generally speaking, the input pattern can be one of
S, P, O, SP, SO, PO, SPO(order doesn't matter because you use an object). If you elect an index to query for each and every pattern, you will end up with 5 indices instead of 6. Please, try yourself to confirm.A last word on the undocumented feature
preferiteIndexOSPanymore;preferIndexorpreferredIndex.Thanks for levelgraph and please, let me know.