You can generate all figures in json, provn, rdf, xml or dot format. Simply run the provided figures.py script. For a quick setup use the provided requirements.txt to install all required packages.

pip install -r requirements.txt

The provenance model for GitLab2PROV provenance graphs consists of multiple submodels, that are concerned with various types of interactions that users can have with a GitLab project aswell as with the git repository contained within the project. A few models have been compiled without prior examples others are derivations of related projects such as git2prov or github2prov.

In total, the GitLab2PROV provenance model includes the following submodels:

1. Git: Addition of a File
2. Git: Modification of a File
3. Git: Deletion of a File
4. GitLab: Commit Web Resource
5. GitLab: Issue Web Resource
6. GitLab: Merge Request Web Resource
7. GitLab: Release & Tag Resource

This document contains a brief explanation for each model. This includes, but is not limited to, a reference table for each PROV element of a model that defines which attributes are attached to the element. Reference tables for qualified relations, i.e. relations that with attached attributes, are also provided.

This document uses the Cypher query language notation to denote relationships/relations. The following ASCII art based notation represents a directed relation r of type R between the vertices S and T.

(S)-[r:R]->(T)

This model captures the addition of a new file to the git repository of a GitLab project by a git commit.

The model includes all human actors involved in the process. In this case these actors are the author and the committer of the git commit represented as agents in the model. The author represents the user that originally wrote the code contained in the commit. The committer represents the user that committed the code on behalf of the author. Committer and author can be the same person but do not have to be.

The commit aswell as all of its parents are captured as activities. Each commit is said to be informed by its parent commit, as each commit builds upon the git repository that the parent commits left behind. The commit is associated to both author and committer as these are the actors responsible for its existance.

Two entities are created for the file that was added in the commit. One, the File entity, represents the origin of the added file aswell as the concept of its originality. The second entity, called FileRevision, represents the added file at the time of its addition. The revision are marked as a specialization of the file origin. Both entities are generated by the commit activity. Both entities are attributed to the author of the commit, the actor responsible for their content and creation.

Author

Committer

Commit

File

File Revision

Some PROV relations in this model are "qualified" relations. In simple terms: Some PROV relations have attributes attached to them. The following tables define the attributes attached to these relations.

File - [wasGeneratedBy] -> Commit

File Revision - [wasGeneratedBy] -> Commit

Commit - [wasAssociatedWith] -> Author

Commit - [wasAssociatedWith] -> Committer

This model captures the modification of a file from the git repository of a GitLab project by a git commit.

The model includes all human actors involved in the process. In this case these actors are the author and the committer of the git commit represented as agents in the model. The author represents the user that originally wrote the code contained in the commit. The committer represents the user that committed the code on behalf of the author. Committer and author can be the same person but do not have to be.

The commit aswell as all of its parents are captured as activities. Each commit is said to be informed by its parent commit, as each commit builds upon the git repository that the parent commits left behind. The commit is associated to both author and committer as these are the actors responsible for its existance. The commit uses the PreviousFileRevision to generate a new revision that accounts for the modifications included in the commit.

Three entities are created for the modified file. One, the File entity, represents the origin of the file aswell as the concept of its originality. The File entity will already exist, due to it being created in the 'Git: Addition of a File' model. The second entity, called PreviousFileRevision, represents the latest file revision of the modified file before modification. The third entity, called FileRevision, represents the revision of the file after the modification has been accounted for. The FileRevision is said to be derived from the previous revision of the modified file. The FileRevision is generated by the commit activity. All revisions are marked as specializations of the File entity.

Author

Committer

Commit

File

File Revision

Previous File Revision

Some PROV relations in this model are "qualified" relations. In simple terms: Some PROV relations have attributes attached to them. The following tables define the attributes attached to these relations.

Commit - [used] -> Previous File Revision

File Revision - [wasGeneratedBy] -> Commit

Commit - [wasAssociatedWith] -> Author

Commit - [wasAssociatedWith] -> Committer

This model captures the deletion of a file from the git repository of a GitLab project by a git commit.

The model includes all human actors involved in the process. In this case these actors are the author and the committer of the git commit represented as agents in the model. The author represents the user that originally wrote the code contained in the commit. The committer represents the user that committed the code on behalf of the author. Committer and author can be the same person but do not have to be.

The commit aswell as all of its parents are captured as activities. Each commit is said to be informed by its parent commit, as each commit builds upon the git repository that the parent commits left behind. The commit is associated to both author and committer, as these are the actors responsible for its existance.

Two entities are created for the deleted file. One, the File entity, represents the origin of the file aswell as the concept of its originality. The second, the FileRevision entity, represents the revision of the file at the point of its deletion. The revision is invalidated by the commit that deletes / removes it from the repository. The deleted revision is marked as a specialization of the original File entity.

Author

Committer

Commit

File

File Revision

Some PROV relations in this model are "qualified" relations. In simple terms: Some PROV relations have attributes attached to them. The following tables define the attributes attached to these relations.

Commit - [wasAssociatedWith] -> Author

Commit - [wasAssociatedWith] -> Committer

File Revision - [wasInvalidatedBy] -> Commit

This model captures the creation and annotation of a GitLab commit web resource, i.e. the webpage of a git commit as displayed by GitLab.

GitLab creates a webpage for a commit as soon as the commit is pushed to the GitLab remote. Users can interact with the webpage by, among other interactions, leaving a comment in the comment section. GitLab captures some of these interactions and stores them in internal data structures. Comments written by users are therefore retrievable through the GitLab API. Retrievable interactions such as comments are considered to "annotate" the web resource.

The model includes all human actors involved in the process of the creation or annotation of a GitLab commit web resource. In this case these actors are the author of the GitLab commit web resource aswell as all users responsible for annotations. The author represents the user that pushed the commit to the GitLab remote and consequently triggered the creation of the web resource. An annotator is a user that is responsible for the existance of an annotation such as a comment. In case of the annotation being a comment, the responsible annotator would be the author of the comment.

The creation of the web resource is captured as an activity. The creation activity is informed by the corresponding git commit that triggered the creation of the commit web resource. The creation activity is associated with the user that pushed the commit to the GitLab remote. In the context of the creation activity, this user is called "Gitlab Commit Author". Each annotation is captured as an activity that uses the latest version of the web resource to generate a new one. Each annotation is associated with the user that is responsible for creating it. Each annotation is informed by either the annotation that precedes it or - if no annotations have been recorded so far - the creation activity. The annotations form a chain of events, that corresponds to the chain of interactions between users and the GitLab commit web resource.

The commit web resource is captured by multiple entities. One for the original web resource and its concept of originality called "GitLab Commit". A second one for the version of the GitLab commit web resource at the time of its creation, called "Commit Version". One entity per annotation capturing the commit web resource right after the annotation happened, called "Annotated Commit Version". The original web resource and the resource version at the point of creation is generated by the creation activity. The original web resource and the first resource version are attributed to the gitlab commit author. Each annotated commit version is generated by the corresponding annotation activity. Each annotated commit version is attributed to its annotator.

Gitlab Commit Author

Annotator

Git Commit

Creation

Annotation

The set of attributes for annotations can change according to the annotation type. Annotations of type "assign_user" carry the attribute "user_name" which stores the name of the assigned user. Other annotation types may have different additional attributes. All recognized annotation types are listed in the "Annotations" section of this document.

Commit

Commit Version

Annotated Commit Version

Some PROV relations in this model are "qualified" relations. In simple terms: Some PROV relations have attributes attached to them. The following tables define the attributes attached to these relations.

Creation - [wasAssociatedWith] -> Gitlab Commit Author

Annotation - [wasAssociatedWith] -> Annotator

Commit - [wasGeneratedBy] -> Creation

Commit Version - [wasGeneratedBy] -> Creation

Annotated Commit Version - [wasGeneratedBy] -> Creation

Annotated Commit Version - [used] -> Creation

This model captures the creation and annotation of a GitLab issue web resource, i.e. the webpage of an issue as displayed by GitLab.

GitLab provides an issue tracker which is accessable through the GitLab API. The GitLab: Issue Web Resource model is structurally similar to the GitLab: Commit Web Resource model. The idea behind it is very similar aswell.

The model includes all human actors involved in the process of the creation or annotation of a gitlab issue web resource. In this case the actors are the author of the gitlab issue web resource aswell as all users responsible for annotations. The issue author represents the user that opened/created the issue in the first place. An annotator is a user that is responsible for the existance of an annotation such as a comment, label, etc. For example: In case of the annotation being a comment, the responsible annotator would be the author of the comment.

The creation of the web resource is captured as an activity. The creation activity is associated with the user that opened/created the issue. In the context of the creation activity, this user is called "Issue Author". Each annotation is captured as an activity that uses the latest version of the web resource to generate a new one. Each annotation is associated with the user that is responsible for creating it. Each annotation is informed by either the annotation that precedes it or - if no annotations have been recorded so far - the creation activity. The annotations form a chain of events, that corresponds to the chain of interactions between users and the gitlab issue web resource.

The issue web resource is captured by multiple entities. One for the original web resource and its concept of originality called "Issue". A second one for the version of the gitlab issue web resource at the time of its creation, called "Issue Version". One entity per annotation capturing the issue web resource right after the annotation happened, called "Annotated Issue Version". The original web resource and the resource version at the point of creation is generated by the creation activity. The original web resource and the first resource version are attributed to the gitlab issue author. Each annotated issue version is generated by the corresponding annotation activity. Each annotated issue version is attributed to its annotator.

Issue Author

Annotator

Creation

Annotation

The set of attributes for annotations can change according to the annotation type. Annotations of type "assign_user" carry the attribute "user_name" which stores the name of the assigned user. Other annotation types may have different additional attributes. All recognized annotation types are listed in the "Annotations" section of this document.

Issue

Issue Version

Annotated Issue Version

Some PROV relations in this model are "qualified" relations. In simple terms: Some PROV relations have attributes attached to them. The following tables define the attributes attached to these relations.

Creation - [wasAssociatedWith] -> Issue Author

Annotation - [wasAssociatedWith] -> Annotator

Issue - [wasGeneratedBy] -> Creation

Issue Version - [wasGeneratedBy] -> Creation

Annotated Issue Version - [wasGeneratedBy] -> Annotation

Annotation - [used] -> Issue Version

This model captures the creation and annotation of a GitLab merge request web resource, i.e. the webpage of a merge request as displayed by GitLab.

The GitLab: Merge Request Web Resource model is structurally similar to the GitLab: Commit Web Resource model. The idea behind it is very similar aswell.

The model includes all human actors involved in the process of the creation or annotation of a gitlab merge request web resource. In this case the actors are the author of the gitlab merge request web resource aswell as all users responsible for annotations. The issue author represents the user that opened/created the merge request in the first place. An annotator is a user that is responsible for the existance of an annotation such as a comment, label, etc. For example: In case of the annotation being a comment, the responsible annotator would be the author of the comment.

The creation of the web resource is captured as an activity. The creation activity is associated with the user that opened/created the merge request. In the context of the creation activity, this user is called "Merge Request Author". Each annotation is captured as an activity that uses the latest version of the web resource to generate a new one. Each annotation is associated with the user that is responsible for creating it. Each annotation is informed by either the annotation that precedes it or - if no annotations have been recorded so far - the creation activity. The annotations form a chain of events, that corresponds to the chain of interactions between users and the gitlab merge request web resource.

The merge request web resource is captured by multiple entities. One for the original web resource and its concept of originality called "Merge Request". A second one for the version of the gitlab merge request web resource at the time of its creation, called "Merge Request Version". One entity per annotation capturing the merge request web resource right after the annotation happened, called "Annotated Merge Request Version". The original web resource and the resource version at the point of creation is generated by the creation activity. The original web resource and the first resource version are attributed to the gitlab merge request author. Each annotated merge request version is generated by the corresponding annotation activity. Each annotated merge request version is attributed to its annotator.

Merge Request Author

Annotator

Creation

Annotation

The set of attributes for annotations can change according to the annotation type. Annotations of type "assign_user" carry the attribute "user_name" which stores the name of the assigned user. Other annotation types may have different additional attributes. All recognized annotation types are listed in the "Annotations" section of this document.

Merge Request

Merge Request Version

Annotated Merge Request Version

Some PROV relations in this model are "qualified" relations. In simple terms: Some PROV relations have attributes attached to them. The following tables define the attributes attached to these relations.

Creation - [wasAssociatedWith] -> Merge Request Author

Annotation - [wasAssociatedWith] -> Annotator

Merge Request - [wasGeneratedBy] -> Creation

Merge Request Version - [wasGeneratedBy] -> Creation

Annotated Merge Request Version - [wasGeneratedBy] -> Annotation

Annotation - [used] -> Merge Request Version

This model captures the creation of gitlab releases and tags.

The model includes all human actors involved in the process of the creation of a release of tag. In this case the actors are the author of the release, the author of the tag and the author of the git commit that pushed the tag.

The creation of a release is captured as an activity called "Release Creation". The creation of a tag is captured as an activity called "Tag Creation". The creation of a commit is captured as an activity called "Commit Creation". The release creation is associated with the author responsible for the release. The tag creation is associated with the author responsible for the tag. The commit creation is associated with the git commit author.

An entity is created for each release, tag and commit. The tag and release entities are considered to be a prov:Collection. Each gitlab release includes multiple asset files aswell as a release evidence file. Each such file is captured in an entity. The release is generated by the release creation activity. The tag is generated by the tag creation activity. The commit is generated by the commit creation activity.

Asset

Evidence

Commit

Tag

Release

Commit Author

Tag Author

Release Author

Commit Creation

Tag Creation

Release Creation

Some PROV relations in this model are "qualified" relations. In simple terms: Some PROV relations have attributes attached to them. The following tables define the attributes attached to these relations.

Release Creation - [wasAssociatedWith] -> Release Author

Tag Creation - [wasAssociatedWith] -> Tag Author

Commit Creation - [wasAssociatedWith] -> Commit Author

Release - [wasGeneratedBy] -> Release Creation

Tag - [wasGeneratedBy] -> Tag Creation

Commit - [wasGeneratedBy] -> Commit Creation

GitLab displays annotations that occur on resources on the webpages of the respective resources. For example, if a resource was mentioned in the comment thread of another resource, this mention is displayed in the comment section of the mentioned target.

These annotations can be parsed from multiple sources that are provided by the official GitLab API. Sadly there is no dedicated endpoint for all annotations that are of interest. Especially annotations that connect resources are difficult to get. Here a quick summary of what data needs to be retrieved, how to parse it and the workarounds that we deployed to achieve annotation parsing.

For label events we use the official API endpoint from which we parse the appropriate annotations ("add_label", "remove_label"). Emoji awards can be retrieved from the appropriate API endpoint. We parse everything else - such as mentions, time tracking stats, due dates, TODO's, etc. - from system notes that GitLab uses to display annotations in their web-interface.

System notes include a string that describe the annotation that they represent. We classify the annotation that the string denotes using regular expressions. If necessary we include named groups in the regular expressions to extract relevant information from the annotation strings. These are later added to PROV element attributes.

Noted, this is not optimal as older GitLab versions employ different string notations for the same annotation. Sometimes only differing by a few characters and other times having a completly different string for the same annotation. In addition there is a problem when parsing imported projects. For example, while parsing a project that was imported from SVN, relevant annotations wheren't recorded as system notes but rather as normal notes. This is not accounted for and is - as of right now - not covered by the current note parsing approach.

Here a list of annotations that we are currently able to parse with a short description of what the annotation is and the API resource from which we parse that annotation.