eSSIF-Lab framework issueshttps://gitlab.grnet.gr/essif-lab/framework/-/issues2020-09-30T08:30:29Zhttps://gitlab.grnet.gr/essif-lab/framework/-/issues/4'Edit this page' links do not work - can we replace it with 'date of last edit'?2020-09-30T08:30:29ZRieks Joosten'Edit this page' links do not work - can we replace it with 'date of last edit'?We currently have three pages that are published through https://essif-lab.pages.grnet.gr/framework/, each having the text 'Edit this page' at their bottom, all of which link to a different (and wrong) location:
- [Introduction](https:...We currently have three pages that are published through https://essif-lab.pages.grnet.gr/framework/, each having the text 'Edit this page' at their bottom, all of which link to a different (and wrong) location:
- [Introduction](https://essif-lab.pages.grnet.gr/framework/docs/introduction) links to facebook (https://github.com/facebook/docusaurus/edit/master/website/docs/introduction.md)
- [Vision](https://essif-lab.pages.grnet.gr/framework/docs/vision-and-purpose) links to facebook 'page not found' (https://github.com/facebook/docusaurus/edit/master/website/docs/essif-lab-vision-and-purpose.md) and
- [Functional Architecture](https://essif-lab.pages.grnet.gr/framework/docs/functional-architecture) also links to facebook 'page not found' (https://github.com/facebook/docusaurus/edit/master/website/docs/essif-lab-functional-architecture.md)
Can this be correctedFoteinos Mergoupis-AnagnouFoteinos Mergoupis-Anagnouhttps://gitlab.grnet.gr/essif-lab/framework/-/issues/12Authoring terminology-related files (i.e. for Terms, Concepts, Patterns, Scop...2021-08-26T13:17:35ZRieks JoostenAuthoring terminology-related files (i.e. for Terms, Concepts, Patterns, Scopes and Glossaries)In this issue, which unifies (and hence closes) issues #7, #8, #9, we specify the requirements that authors of terminology-related files must adhere to. This top message contains these requirements. Any updates to the requirements will l...In this issue, which unifies (and hence closes) issues #7, #8, #9, we specify the requirements that authors of terminology-related files must adhere to. This top message contains these requirements. Any updates to the requirements will lead to this top message to be edited/updated, so that it always contains the latest state of affairs. Subsequent messages document the discussions.
We currently have the following terminology related items:
- a **scope** is something within which concepts can be associated with terms for a specific purpose, thereby creating a vocabulary that can be used to meaningfully express ideas, arguments, etc. so as to realize that purpose. For the sake of this issue, a scope serves the same purposes as a namespace.
- a **pattern** captures a set of concepts, relations between them, and rules or constraints that (instances) thereof comply with. As such, it is a concise and possibly formal description of a coherent set of ideas, a mental model if you will, that can be used to facilitate one's thinking about/with these concepts. A pattern must be defined within a scope.
- a **concept** tries to capture the idea behind a classification of entities, allowing one to reason about everything in the class as if it were one thing. It MUST be defined within a scope, and in conjunction with a term that is used to refer to instances of the concept in that scope.
- a **term** is a word or phrase that is used in at least one scope to refer to some concept; it is a 'link' between a scope and a concept, allowing each scope to define its own terms, yet build on the concepts developed by others. Obviously, terms too MUST be defined in a scope, as a reference to a concept (possibly, but not necessarily, from another scope).
- a **glossary** is a list of terms with (short) explanations, usually aimed to help people understand texts around a certain (set of) topic(s) in some context(s). A glossary MUST be defined in a scope.
We have a template file for every such item
- **For TEv1**,
- templates can be found in `docs/terminology-engine-v1-templates`
- you should only use the templates for a concept, term, or pattern (for the other stuff TVv1 has no generators).
- copy the template to the `docs/terms` directory, rename it in an appropriate fashion, then edit the header (instructions are included in the template), and add the content (preferably according to the structure provided in the template).
- **For TEv2**
- templates can currently be found in `doc-sources/terminology/templates`.
- we will describe their use when TEv2 gets underway.Foteinos Mergoupis-AnagnouFoteinos Mergoupis-Anagnouhttps://gitlab.grnet.gr/essif-lab/framework/-/issues/14Terminology Engine v2 specifications2021-08-26T13:17:35ZRieks JoostenTerminology Engine v2 specifications# Introduction
Terminology Engine v1 (TEv1) is now operational; it enables us to not only define terminology, but also to actually put it to use and help readers of documentation understand what they read, and drill down to level of unde...# Introduction
Terminology Engine v1 (TEv1) is now operational; it enables us to not only define terminology, but also to actually put it to use and help readers of documentation understand what they read, and drill down to level of understanding that they seek (some are satisfied with the popups that appear on defined terms, others will click on the term to get further context and backgrounds). That is: after the existing documentation has been updated to take the full advantage of this tool.
Two of the major objectives of the eSSIF-Lab project include (a) the creation of an ecosystem of developers (and users) that continues to thrive after the project has died, and (b) ensure interoperability, which is not just about the technology, but about the documentation as well.
Given that different parties (subgrantees) will have use different vocabularies/terminologies, we seek for ways to ensure terminological interop while accommodating each party's specific terms.
# Purpose
The main objective of TEv2 would then be to facilitate the new version of the eSSIF-Lab architecture, that is going to be constructed by eSSIF-Lab consortium partner(s), every subgrantee of the infrastructure call, and each subgrantee of BOC#1 that produces (open source) components that extend the infrastructure. The resulting architecture should contain a functional description of each of these components, and this description must be embedded in the 'integral whole'.
# High-level requirements
The high-level requirements of TEv2 are:
- [ ] Writing terminology and other documentation shall be easy for different authors. This means that they must not be required to go through an extensive learning-cycle. Also, they should be provided tooling that helps them to write documentation that passes the CI-pipe without any problems, specifically by not only pointing out whether or not they made a mistake, but as much as possible by identifying where, in their own documents, the cause of the error lies, and what it takes to fix it.
- [ ] It should be possible for authors to keep their documentation (including term-definitions) separate from that of other subgrantees (in a 'scope' of their own), so that they can document their stuff in an internally consistent/coherent way without being bothered by terms that others use. This scope may also serve as a namespace. Ideally, they should be able to write their documentation irrespective of the implementation of the TEv2 (meaning that it would work equally well if TEv2 were implemented as a Docusaurus extension or as a stand-alone tool (that e.g. could update a Confluence or wiki site).
- [ ] It shall be possible to work with multiple such scopes (see the terminology-pattern when that becomes available). A scope has its own vocabulary, which consists of the terms (and other stuff) defined within that scope, supplemented with terms that are defined in other scopes/namespaces (where the term itself may be changed if that is appropriate for that scope).
- [ ] It shall be possible for authors to use terms in their documentation files that have been defined within the scope in which that file is created, where 'using a term' means that it is made to stand out when rendering the document, that it has a popup, and that clicking on it navigates the user to the place where it is defined (as in TEv1). It shall also be possible to use terms that are defined in another scope, provided that the scope supports TEv2 functionality.
- [ ] For existing terminologies, such as that of NIST, DIF, Sovrin, etc., that do not support TEv2 functionality, we may decide to create a scope within the eSSIF-Lab context that does support TEv2 functionality, thus allowing authors to refer to such terms with all the benefits that TEv2 brings.
- [ ] It shall be possible (for technically sufficiently capable engineers), to create generators that take one or more source file(s) from documentation authors, and create artifacts from that, e.g. for making dictionaries, concept-graphs, statistics and more. Note that this item only requires that such engineers be facilitated to create such generators. Nevertheless, for maintainability, it is suggested that every scope should contain a directory `generator-specs`, that contain the files that generators use to generate their artifacts. Such specification files would conform to the generic TEv2-document-template described above, where their `type`-attribute would correspond with the generator. So a file that would specify an artifact to be generated by e.g. the xxx-generator would specify `xxx` in its `type` field (so `glossary` and `dictionary` would be valid `xxx`s, but we can think of (but not necessarily implement more).
TEv2Foteinos Mergoupis-AnagnouFoteinos Mergoupis-Anagnouhttps://gitlab.grnet.gr/essif-lab/framework/-/issues/20Topics for consideration of long-term Terminology Engine developments2020-10-09T09:07:14ZRieks JoostenTopics for consideration of long-term Terminology Engine developmentsOver the last days, as an author, I experienced some difficulties in writing terminology-files. To me, it is not clear which of these difficulties should/could be resolved, and which is just something we'll have to deal with (or defer to...Over the last days, as an author, I experienced some difficulties in writing terminology-files. To me, it is not clear which of these difficulties should/could be resolved, and which is just something we'll have to deal with (or defer to vsn _n_ of the terminology engine). I created this issue so that I can log the difficulties I run into, and we can discuss what to do with it (if anything), and when.
I think we can live with these issues - at least for now. I can imagine that at some later stage (perhaps even after TEv2) more author-oriented parser/content processors might be constructed for use in a CI/CD pipeline that would resolve this issue, and they might have associated tools (e.g. vscode extensions) that help authors identify and correct any mistakes.
### 1. Referencing to pictures and documentation files that are not part of the terminology corpus
Currently, authors need to be aware of the underlying software, in our case docusaurus when making such references. Drawbacks include:
- the documentation corpus is not trivially transferrable to other systems (problems may include referring to other documents, or specifics in the markup);
- constraints for writing documents come from multiple sources (that may be independently updated from each other).
The implication might be that a parser/generator be developed that more or less replaces docusaurus, at least many of its features, which seems to be an overkill.
### 2. Docusaurus peculiarities
While I very much like the docusaurus idea of having a header with key-value pairs that can be used for further processing, the body of such files is very much code-writer-oriented. For example, including an image requires the statement `import useBaseUrl from '@docusaurus/useBaseUrl';`, allowing the author to subsequently include the image `somefile.png` that has been stored in 'static/images/' using HTMLtags:
~~~
<img
alt="some alt text"
src={useBaseUrl('images/somefile.png')}
/>
~~~
Thinking in terms of our terminology model, it might be beneficial to treat an image in a similar way as a term (they're both semiotic [signs](https://en.wikipedia.org/wiki/Sign_(semiotics)), which would imply that they are stored in the location of the scope in which they have been defined.
## 3. Generator feedback to authors.
I've seen the following happen:
![image](/uploads/3b3606ee7073ba26b3be78620c2b676e/image.png)
It is hard for (authors like) me to find out what is wrong and how to correct this. Better ways of informing users would be welcome. If the Terminology Engine is to be productized, better support of its users in terms of identifying mistakes and helping them fix it seems a basic requirement.
https://gitlab.grnet.gr/essif-lab/framework/-/issues/21Change request - Glossaries and Pattern lists2020-10-29T15:01:59ZRieks JoostenChange request - Glossaries and Pattern listsCurrently, the glossary is generated by processing all files in the `doc/terms` directory.
However, glossaries are supposed to list terms, whereas the `doc/terms` directory also contains other artifacts, such as pattern-descriptions.
T...Currently, the glossary is generated by processing all files in the `doc/terms` directory.
However, glossaries are supposed to list terms, whereas the `doc/terms` directory also contains other artifacts, such as pattern-descriptions.
This issue makes the following requests:
### generate a glossary
that
- [x] **only includes files where the `type` attribute in the docusaurus header is either `concept` or `term`**. That not only rids us from the 'undefined' entry in the glossary (which links to a README), but it also allows us to have other kinds of documents in that directory, e.g. terminology-maintenance scripts for editors (e.g. VSCode). AND, it would get rid of a bug, which is that the glossary seems to be generated based on filenames rather than the `id` attribute (evidence of which is that if they differ, an error is generated that says there is a broken link in the glossary, which cannot happen if the `id` attribute would have been used).
- [x] **uses (the (new) 'glossaryText' header item rather the 'hoverText' header item)**, and processes the %%-references that appear in the 'glossaryText' header item in the same way as it is used in other texts.
- [ ] **appends the generated text to the `docs/glossary.md` file**; note that later, when we allow for multiple glossaries, a glossary-file should provide additional specifications that will change the glossary generation process.
### generate a pattern-list,
with similar requirements, i.e.: that only includes files where the `type` attribute in the docusaurus header is `pattern`, allows for %%-references in the text for the various patterns in the list, and is appended to `patterns-list.md` file.
CC: @fmerg, @miliadi, @kathanTEv2https://gitlab.grnet.gr/essif-lab/framework/-/issues/23TEv2 - generic document structure2020-10-30T11:52:12ZRieks JoostenTEv2 - generic document structureThis issue specifies the structure of documents that are processable by TEv2 and/or its additions (generator-plugins). When approved, the text may become part of the eSSIF-Lab documentation that describes its terminology management.
# H...This issue specifies the structure of documents that are processable by TEv2 and/or its additions (generator-plugins). When approved, the text may become part of the eSSIF-Lab documentation that describes its terminology management.
# Header
Every file starts with a docusaurus header, i.e.
```md
---
id: document-identifier
title: title that will show when the document is rendered
... (other [docusaurus header items](https://v2.docusaurus.io/docs/markdown-features/#markdown-headers))
---
```
## TEv2 specific header attributes
After the regular docusaurus header attributes, additional attributes must be specified that allow TEv2 to properly function. These attributes must be in the same header block as the docusaurus attributes (i.e. in the same block delimited by the `---` lines), as follows:
```md
---
... (docusaurus attributes)
scopeid: <identifier for the scope in which the document is defined>
type: <type of the document>
typeid: <identifier of the individual document (given the values of `scopeid` and `type`)>
---
```
where:
- ***scopeid*** (required) identifies the scope within which the document (and its contents) belongs, and is being defined/updated/...
- ***type*** identifies the kinds of contents that can be expected in the document. The following types are being used (and should be supported):
- **scope** the file contains the specification/description of a scope.
- **concept** the file contains the definition of a term and a description of the concept to which it refers;
- **term** the file contains the definition of a term and a reference to a concept;
- **glossary** the file contains a specification for the generation of a glossary that is to become part of the same scope to which the glossary-file belongs;
- **glossary** the file contains a specification for the generation of a dictionary that is to become part of the same scope to which the glossary-file belongs;
Additional types may be defined as necessary.
- ***typeid***: text that ensures the triple (scopeid,type,typeid) always identifies the document.
## TEv2 Generator-specific attributes
Additional attributes may be added for purposes defined by different generators.
For example:
- `hoverText: <text that pops up when the user hovers over a document-reference>` is used by the documentation website generator, for the purpose of producing a popup when the user hovers over a reference to that document. This text may not include %%-references.
- `glossaryText: <text>` is used by the glossary-generator as the explanation of a term. This text MAY contain %%-references where the `|` MUST be replaced by the `^` character (as the `|` character causes errors when it is part of a docusaurus-header).
- `stage: <stageid>` is foreseen to be used in ToIP terminology life cycle management processes.
Attributes that are not used by Docusaurus, TEv2 or a generator will be ignored (so you can leave them there if you don't use them).TEv2https://gitlab.grnet.gr/essif-lab/framework/-/issues/24TEv2 ways of referencing documents2020-10-13T13:16:32ZRieks JoostenTEv2 ways of referencing documents
# TEv1 way of referencing
In TEv1, documents are referable by the docusaurus mechanism for referencing, i.e. by using the docusaurus `id` attribute for referencing, optionally pre-pended by a (relative) path to the dictionary in which ...
# TEv1 way of referencing
In TEv1, documents are referable by the docusaurus mechanism for referencing, i.e. by using the docusaurus `id` attribute for referencing, optionally pre-pended by a (relative) path to the dictionary in which the referred-to document lives. For example, a document may refer to a document with attribute `id: some-id` by writing `%%shown text|some-id%%`, or `%%shown text|./some-id%%`.
This mechanism works for terminologies under the assumption that there's only one terminology/scope to deal with.
# TEv2 ways of referencing
In TEv2 we need a referencing mechanism that allows us to deal with multiple scopes (namespaces).
Now follows a suggestion for allowed syntax. In this suggestion, when we say 'in the same scope' or 'of the same type', this means 'same as: the document that contains the reference'. The idea of the syntax is that after selecting all documents that satisfy the specified selection criterion, only one document remains. If this is not the case, an error condition exists that should be 'thrown' to the author with a message that allows him to determine what went wrong AND how to fix it.
- [ ] `%%shown-text|<typeid>%% selects all documents with attribute `typeid: <typeid>` that are in the same scope.
- [ ] `%%shown-text|<type>-<typeid>%% selects all documents with attributes `type: <type>` AND `typeid: <typeid>` that are in the same scope.
- [ ] `%%shown-text|<scopeid>/<typeid>%% selects all documents with attributes `scopeid: <scopeid>` AND `typeid: <typeid>`.
- [ ] `%%shown-text|<scopeid>/<type>-<typeid>%% selects all documents with attributes `scopeid: <scopeid>` AND `type: <type>` AND `typeid: <typeid>`.
Notes:
- this syntax can be extended in future versions by replacing the `<scopeid>` in references by a 'path' of (nested) scopes, and perhaps even by an URI+'path-of-nested-scopes'.
- a scopid is NOT a directory(name) per se. It SHOULD be possible to have a mapping between a (nest of) scopes and arbitrary directories, e.g. by means of some JSON object such as
~~~json
{ "scope-register": [
{ "scopeID": "essifLab"
, "scopename": "eSSIF-Lab"
, "location": "https://gitlab.grnet.gr/essif-lab/framework/terminology"
, "subscopes": [
{ "scopeID": "essifLabTerminology"
, "scopename": "eSSIF-Lab Terminology"
, "location": "https://gitlab.grnet.gr/essif-lab/framework/terminology/terminology"
}]
}]
}
~~~
Such a JSON object may be generated from scope-files (.md files that contain scope-data), or by having the admin of the system maintain that JSON object.
# TEv2 shorthand references
When writing this text, more than 71% (216/301) of the references in the eSSIF-Lab documentation were of the form `%%shown-text|reference-text%%` where `shown-text`, after converting it to all lower-case, would equal `reference-text`. This percentage rises to over 83% (252/301) if also spaces in `shown-text` were converted to `-` characters. This makes the case for specifying that `%%shown-text%%, provided it only consists of characters {[A-Z], [a-z], '-', ' '}, must be rewritten to `%%shown-text|reference-text%%` prior to further processing, where `reference-text` is the all-lowercase version of `shown-text` where every space is replaced by `-` character.
If `shown-text` contains characters other than those allowed, an error conditions exists (which must be thrown to the author etc.)TEv2https://gitlab.grnet.gr/essif-lab/framework/-/issues/25%%-references pre-processing2020-10-19T14:12:56ZRieks Joosten%%-references pre-processingWhen writing documentation, it frequently occurs that I make some mistakes that might be resolved by some (pre)processing of %%-references. Also, it frequently occurs that I have the same text before and after the `|` separator.
To mak...When writing documentation, it frequently occurs that I make some mistakes that might be resolved by some (pre)processing of %%-references. Also, it frequently occurs that I have the same text before and after the `|` separator.
To make it much easier for authors, I suggest we do the following:
0. Assume we have `text` as the entire text between `%%`-markers (so excluding such markers)
1. if `text` contains the `|`-character, then `showText` is the part before, and `refText` is the part behind that character.
2. else, `showText = text` and `refText = text`.
3. make `refText all lowercase, and replace every space(s), i.e. `/\s+/g`-sequences, with a single `-`-character.
4. if the file to which the resulting `reftext` refers does not exist, it would be nice to have this condition logged (and presented at the end of processing), preferably with the filename, line and columnnumbers of the `%%`-structure that caused the event.
I have tried to do this (except for point 4) myself, but I keep getting `Unhandled promise rejection` errors, and also (but less frequently) `ReferenceError: useBaseUrl is not defined` errors.TEv2https://gitlab.grnet.gr/essif-lab/framework/-/issues/26TEv2 - including images (perhaps also other kinds of files)2021-05-11T06:37:23ZRieks JoostenTEv2 - including images (perhaps also other kinds of files)When editing our documentation, I found that having `<img`-constructs in documentation files may be convenient for seasoned (HTML) writers, but for the casual documentation editor that knows WHAT (but isn't well-versed in HOW) to write h...When editing our documentation, I found that having `<img`-constructs in documentation files may be convenient for seasoned (HTML) writers, but for the casual documentation editor that knows WHAT (but isn't well-versed in HOW) to write his stuff, it is not. I also found that as I was making some serious changes throughout the entire documentation, %%-notation stuff ended up in `alt`-texts, which the compiler cannot handle.
This issue exists so as to accumulate experience for deciding whether or not to introduce some (simple) syntax for including images (perhaps also: other (text)files).TEv2https://gitlab.grnet.gr/essif-lab/framework/-/issues/27Feature request - test for illegal use of %%-references2021-08-16T14:14:21ZRieks JoostenFeature request - test for illegal use of %%-referencesI recently had quite some trouble debugging .md files that caused `docker-compose up --force-recreate --build` to produce errors while building the client (whatever that is). @miliadi pointed me in the direction for solving it (on [Slack...I recently had quite some trouble debugging .md files that caused `docker-compose up --force-recreate --build` to produce errors while building the client (whatever that is). @miliadi pointed me in the direction for solving it (on [Slack](https://essif-lab.slack.com/archives/G01BGHF6XQC/p1629106669011100)). While this fixed one problem, several others remained but knowing where/what to look for made the difference.
In order to prevent such situations to occur again (in fact, this has happened before but somehow was forgotten, which I do not want to be repeated again), I would like a new feature to be implemented in the CI-pipe that is capable of logging such errors and providing the information a developer needs to correct the markdown.
This could be done as follows:
1. we have a configuration file (located in a directory that can be accessed by the user, e.g. in `docs`) that contains a list of regexes that match illegal syntax. For starters, this list contains the following regexes: `^hovertext:.*?%%` and `(alt|src)\s*=\s*"[^"]*?%%`
2. at an appropriate place in the CI-pipe:
- scan all files that are going to be further processed (i.e. where <Term-constructs can be created), and for each file:
- log any match of that file with any of the regexes in the configured list, where the log states the regular expression that matches, the file in which the match occurred, and the line number and character position of the match.
- after all files have been scanned, throw an error if one or more matches occurred.Maria IliadiMaria Iliadi