Abstracting Data Updates over a Document-oriented interface of a Permissioned Decentralized Environment

Jitse De Smet

Abstracting Data Updates over a Document-oriented interface of a Permissioned Decentralized Environment

Motivation
Research Question and Hypothesis
Proposed Solution
Results
Conclusion

Motivation

Data is the New Gold(?)

Deloitte states data is a Strategic asset

Forbes: How To Make Use Of The New Gold: Data

Imec: Data mogen niet het nieuwe goud zijn

Data is not the new oil say Prof Sir Nigel Shadbolt and Sir Tim Berners-Lee

Data is very valuable, even more so, you often hear that "data is the new gold". I gathered a few articles that state their own opinion on this. We have Deloitte stating that viewing data as a strategic asset is essential and that companies should organize their information system in such a way that the data they gather can be used as fuel. We also see Forbes comparing data to raw materials. They state that data acts as the raw building block for producing digital products. imec then states that data shouldn't be the new gold. Unlike gold, data can be shared by multiple parties at the same time. They therefore suggest that we should create data spaces instead of data silos so multiple people can benefit from data. Prof Sir Nigel Shadbolt and Sir Tim Berners-Lee share this vision stating that data should not be the new oil. Instead, it should be viewed as basic need, just like water. We can thus conclude that yes, data is essential, but that data should not be controlled by massive companies. This not only holds from an economic point of view, but also a social point of view. Where we see that the centralization of data in big centralized silos causes privacy related issues and social turbulence.

Motivation

Decentralization initiatives

Solid
Bluesky
Mastodon
Blockchain based initiatives

Solid

Use Existing Web Technologies
Self Governed Data Store
Heterogeneity
- of Interface
- of Data
- of Structure

Handling Heterogeneity

Research Question and Hypothesis

"How can we abstract data updates over a document oriented interface of a permissioned decentralized environment behind a query abstraction layer ?"

I therefore **ask the question**: "How can we abstract data updates over a document oriented interface of a permissioned decentralized environment behind a query abstraction layer?" Let's investigate that question, in more detail: Abstract data updates - We aim to abstract the query process, so a developer does not need to interact with the pod interfaces directly. Document oriented interface - The interface we interact with exposes data through HTTP documents. (The Solid specification describes the use of such an interface through LDP.) Permissioned - Each HTTP resource has access rights configured, these rights can either grant or deny access to resources for specific users. Decentralized environment - Each pod is self governed and limited rules apply to the system. A loosely defined system allows data publisher to be opinionated Query abstraction layer - We want the abstraction to happen through a declarative query. In this work we use the SPARQL query language but others could be used like GraphQL. We hypothesize that we can create an automated client capable of deciding where to store a resource given a pod. Such a client should have maximum execution time overhead of four times, and maximum double the HTTP requests.

Solution

Heterogeneity of Data → Describe Data
Heterogeneity of Structure → Describe structure

Can use a shape description language like ShEx or SHACL.
Example: SHACL Shape Description of a social media post.


:postShape a sh:NodeShape ;
  sh:property [
    sh:path rdf:type ;
    sh:hasValue ldbc:Post ;
  ] ;
  sh:property [
    sh:path ldbc:creationDate ;
    sh:datatype xsd:dateTime ;
    sh:minCount 1 ;
    sh:maxCount 1 ;
  ] ;
  sh:property [
    sh:path ldbc:id ;
    sh:datatype xsd:long ;
    sh:minCount 1 ;
    sh:maxCount 1 ;
  ] .

Example: LDP Structure

posts/
  |- Valencia
  |  |- #one
  |  |- #two
  |- Ghent/
  |  |- #one
  |  |- #two
  |- Paris/
  |  |- #one
  |  |- #two
  |  |- #three

posts/
  |- 30-01-2024/
  |  |- #one
  |  |- #two
  |- 14-02-2024/
  |  |- #one
  |  |- #two
  |- 17-05-2023/
  |  |- #one
  |  |- #two
  |  |- #three
  |  |- #four

Can use indexes like Type Indexes or Shape Trees.
Example: Shape Trees describing al list of files


<#PicturesTree>
  a st:ShapeTree ;
  st:expectsType st:Container ;
  st:shape ex:PicturesShape ;
  st:contains <#PicturesByCityTree> .

<#PicturesByCityTree>
  a st:ShapeTree ;
  st:expectsType st:Container ;
  st:shape ex:PicturesByCityShape ;
  st:contains <#PictureTree> .

<#PictureTree>
  a st:ShapeTree ;
  st:expectsType st:Resource ;
  st:shape ex:PictureShape .

Is this enough?
To check that, I listed some functional requirements and user stories.
The answer: NO.

To create an automated client, we essentially need to figure out what files hold what data and why. For this, we look back at two heterogeneity that we need to handle. First the heterogeneity of data, we can tackle this by describing our data. In the context of the semantic web, there are two possible ways of doing that, either through ShEx, or through SHACL. We provide an example SHACL shape description of a social media post. The other kind is the heterogeneity of structure. As you can see here, we provide two totally reasonable ways of structuring your social media posts. To describe such in Solid, you can use either Type Indexes or Shape Trees. Shape Trees act like the natural extension of shape descriptions to file structure, specifically to the system used by Solid, namely LDP. We provide an example shape trees example. **Is this enough?**

What are we missing?

What if multiple directories match?
What if no directories match?
How are resources grouped?
Am I creating a leaf directory?
Can I update? Does it stay where it is?
Do all clients follow the same rules?

Introducing: Storage Guidance Vocabulary (SGV)

If I want to insert a resource, I can **see what Shape Tree would match** and put it there, **but...**: 1. What if multiple directories match? 1. Do I duplicate? 2. Is one canonical and the other one links to the resource saved in the canonical? 3. And how do I decide which one is canonical? 2. What if no directories match? 3. How are resources grouped? 1. Can I just infer that picture-by-date example is just that? 2. What if I need to create a new date directory? 4. Is that new directory I created a leaf? 1. Or should I make even more directories? (Can be inferred from Shape Tree) 5. What to do if a resource is changed? 1. Should I alter the Shape Tree? 2. Should I move the resource? 3. Do I have a distance metric, and do I move when the distance is too great? 6. Should all clients abide to the structural information? Click on the SGV link.

Storage Guidance Vocabulary

Resource Collection
Unstructured Collection
Structured Collection
Canonical Collection
Derived Collection
Resource Description
Group Strategy
Store Condition
Update Condition
Client Control

- *Resource Collection*: Corresponds to a group of RDF resources. - *Unstructured Collection*: Corresponds to a classical LDP container or HTTP resource. - *Structured Collection*: A canonical or derived collection. (below) - *Canonical Collection*: A resource collection containing resources. - *Derived Collection*: A resource collection that stores resources already stored by one or more other structured containers. - *Resource Description*: A way of describing resources, for example through ShEx or SHACL. - *Group Strategy*: A description of how resources should be grouped together, for example: my images are grouped per creation date. - *Store Condition*: When multiple collections are eligible to store a resource, the store condition decides what collection(s) actually store the resource. Allowing the creation of a store priority system. - *Update Condition*: Describes what to do when a containing resource is changed. - *Client Control*: Describes the amount of freedom a client has when trying to store a resource.

Storage Guidance Vocabulary

Schematic overview of an SGV creation flow

1. The client gets the SGV description of the storage space (can be cached). 2. The client checks all canonical collections and checks if the resource to be inserted matches a resource description of the collection. 3. If the resource matches a description, the client checks the store condition of the description given the eligible collections. 4. For each collection that stores the resource: 1. The client checks the group strategy of the collection and groups the resource accordingly, deciding on the name of the new resource. 2. The client checks the collections that are derived from this collection. Step 4 is executed for all collections that are derived from this collection, and the resource matches the description. 5. The client performs the store operation. Also reason about the HTTP overhead caused.

Storage Guidance Vocabulary

Schematic overview of an SGV update flow

Taste the pudding

Empirical Evaluation

Create Resource

frag. Strat.	ops/sec	Average Time (ms)	Margin
by date: SGV	22	44582.068	±1.73%
by date: RAW	35	27899.513	±2.07%
one file: SGV	6	149415.739	±2.98%
one file: RAW	7	134361.192	±8.66%
own file: SGV	10	91851.395	±2.56%
own file: RAW	13	76672.217	±3.07%
by location: SGV	23	43005.366	±2.20%
by location: RAW	35	28003.949	±2.53%

Empirical Evaluation

Move Resource

frag. Strat.	ops/sec	Average Time (ms)	Margin
by date: SGV	7	141940.530	±1.28%
by date: RAW	11	87113.119	±0.75%
one file: SGV	2	343690.220	±1.70%
one file: RAW	4	208930.211	±2.04%
own file: SGV	5	177991.908	±0.58%
own file: RAW	12	80729.940	±1.06%
by location: SGV	7	133052.120	±0.60%
by location: RAW	12	81066.196	±1.15%

Conclusion

Automated Client with limited overhead is possible
Lack of server-side control
Inter-pod Updates
Investigate Other Interfaces
Structure has a high influence on execution time
Smart Access Control
CAP / ACID / BASE

* Lack of server-side control - Inherent problem. Since there is no server side control, a client need not follow the SGV description. This means that they can easily break the structure. * Inter-pod Updates - What if I want to move data between pods? As of right now, updates need to happen in one pod. Expanding this mechanism to multiple pods, so we could for example move a resource is interesting and not trivial. For example, can the broad system have two copies of the same resource? So is there an "at least once", "exactly once" or "at most once" existence policy? * Investigate Other Interfaces - Is document oriented the best? Using document oriented interface we basically got the permissions for free because we just have permissions on documents. However, to create an automated update client, SGV is needed just to derive resource locations. Furthermore, they come with some drawbacks too (see [blog post of Ruben Verborgh challenging REST and other interfaces](https://ruben.verborgh.org/blog/2024/05/30/the-webs-data-triad/)) * View Creation and Discovery - Structure has a high influence on execution time We saw that the structure of the storage structure is large factor for execution times, I think it would be interesting to have a server that creates derived resources on the fly when it notices that certain access patterns can increase productivity. * Smart Access control - Now that we know what a document is, can we guess what AC means? I wonder whether when we sufficiently describe resources contained in a document and their heritage, it might be possible to derive what it means to put certain access rights on a document. Such an access control derivation could be usefully for example when one data store is exposed through multiple interfaces. * CAP / ACID - transaction / BASE - CRDTs: I think there is merit in investigating ACID transaction in the context of a Solid pod. Application developers are used to transactions, and we should look at whether we can enable them in the context of Solid. For example, a certain resource could be annotated in such a way that clients know the resource supports transactions. \ In the same way, it also makes sense to investigate CRDTs in order to create fast applications. Both of these systems can be abstracted by a query engine. I think that when we are able to add these systems, developing for Solid will be so easy, developers might even yearn for it. \ It's also important to note that Solid does not have built in partition tolerance, so we have some room for playing around in the CAP space. Note: [Noel De Martin](https://noeldemartin.com/) presented a CRDT vocabulary on the [2nd Solid Symposium](https://events.vito.be/sosy2024). He explains his work on [YouTube](https://www.youtube.com/watch?v=vYQmGeaQt8E). His work is a step in the right direction but lacks logical clocks.

Abstracting Data Updates over a Document-oriented interface of a Permissioned Decentralized Environment

Abstracting Data Updates over a Document-oriented interface of a Permissioned Decentralized Environment

Motivation

Motivation

Solid

Handling Heterogeneity

Research Question and Hypothesis

Solution

What are we missing?

Storage Guidance Vocabulary

Storage Guidance Vocabulary

Storage Guidance Vocabulary

Taste the pudding

Empirical Evaluation

Empirical Evaluation

Conclusion

Time for Questions