dokieli: decentralised authoring, annotations and social notifications

This paper describes the dokieli application for document publishing, authoring and interaction in a decentralized way. After introducing the problem space, the whole architecture and the implementation is presented in detail.

The application captures an interesting topic that is often discussed in the academic world. Further, the architecture and the implementation seems to be well thought out. Especially the decentralization and the easy applicability helps the user to adopt the system.

In my opinion, the paper is rather a detailed documentation of the application than a scientific paper. It contains a lot of information, e.g. captured in the 17 figures, that are interesting for the actual user but they are not necessary in a scientific paper. Further, for me it is not clear how the system really differs from existing works, it is only mentioned that other systems are "lacking facilities for linking discourse". A shorter paper focusing on the main aspects and the differences to other systems would be more valuable for the community.

The paper presents a client-side application (named dokieli) which allows the semantic authoring (creation and publication) as well as the sharing of documents using HTML+RDFa as baseline technology. The paper comprehensively describes the system and tries to mark down the differences to existing, similar systems like [9] and [6].

In general the paper is more a technical report or system documentation than a scientific publication. I think the interesting parts of this work would have made a great short paper, as e.g. some of the figures are not helpful or necessary to understand the application. In addition, I would recommend the authors to clarify a) their contribution in the paper (what is new and what was reused from other applications) and b) the differences to existing systems, as this is only slightly done at the current state.

Minor Remarks:

• Abbr. LDP is used beforehand but mentioned first in 4.5
• Capitalization of Words within titles (e.g. “Modes *of* Operation” or “Semantics *and* Linking”)
• Captions of figures with or without “.”
• Unification of capitalization of keywords within the text “Listing”, “figure”, …
• Figure 11 seems to be missing
• Within the online version of the paper (scarven.ca/dokieli) 4 authors are listed
• Affiliation of Author1 is incomplete (*I*nformation Systems …)

This paper, presenting the architecture of the open source implementation dokieli, is of high quality, well structured, and written in a clear and understandable manner. The implementation demonstrates the high potential of the concepts described in the paper and is an innovative approach for collaborative editing, semantic annotation, and publishing/share structured texts (e.g. articles).

The present work describes a client-side application for "decentralised authoring, annotations and social notifications". It is a particularly relevant work as many academics could benefit from a better tooling support in scholarly writing, debating and publishing.

The paper describes in much detail the many features of the tool. Some requirements include user-friendliness, decentralization, or progressively enhanced functionality. Other ideas presented in the paper (e.g. the annotation of Web pages) have already been published by the same author(s) in the past. These features altogether contrast the proposed architecture from established tools for content editing and publishing like Google Docs or Authorea.

However, despite of the rich feature set, there is no evidence in the paper that the mentioned requirements are fulfilled by the tool. It is described that the application is built on existing tools like a rich editor, and it is complemented by a mechanism that allows to host data on a private datastore for privacy. But an evaluation e.g. in the form of a user study is highly missing to make this a sustainable scientific work.

Compared to previous publications of the author(s), I cannot identify many novelties that make this scientific work sufficiently distinctive. It looks like a natural evolution of previously published research [1] with some additional features. Maybe the main focus should be shifted more towards the core additions, like the possibility to set up personal data spaces with access control, interaction, and notification. Still, the benefit compared to online editing platforms needs to be made clear.

From a readability point of view, it appears that some statements are repeated several times. E.g., 4.5. Data Storage and 4.6 Modes of Operation could be combined into a single heading. Generally I think that the paper could be made much clearer by some simple restructuring. I found also the introduction a bit difficult to read, esp. as it deviates a bit from the common academic writing style.

Provided the limited scientific novelty and the weaknesses outlined above, my recommendation is a weak reject for this paper.

[1] This ‘Paper’ is a Demo, ESWC 2015 Satellite Events.

Meta-review:

Taking into account the balance of other reviews, I recommend rejecting this paper on the following grounds:

1. A need to rewrite the paper to fit a more typical scientific narrative/tone.
2. A need to provide stronger analysis of the relationship to previous/related work.
3. A need to provide greater clarity of the scientific contribution.
4. A need to more clearly demonstrate the delta over previous work.

The outcome aside, I would like to highlight the positive comments of the reviewers on the value of the work, and the likely interest to the community, and would encourage the authors to resubmit to a related conference/workshop or a future LDOW once these issues have been addressed.

interesting. dokieli for decentralised authoring, annotations and social notifications https://twitter.com/csarven/status/697810297080442880

(My review was also posted at https://gitter.im/linkeddata/dokieli?at=56bdd64ffa79226456f9eda8 )

I must admit I agree with the meta-review, I think the reviews are fair in that the article reads more like an Application Note or technical report, and would benefit from a stronger evaluation and justification of its novelty.

I don't think details like the flow diagrams are needed, also the pretty publishing mode pictures feel out of place . The technical work seems sound and very promising - this kind of move towards collaborative and distributed easy-to-use semantic publishing is exactly what is needed to revive academic publication for the 21st century.

I would have appreciated a stronger focus on *provenance* and attribution, e.g. FOAF is mentioned - strong identification of authors should be even more important in a distributed model, so I would have expected some relation to ORCID - which RDF representations provides FOAF and PROV descriptions of people (and currently planning integration with SPAR ontologies).

Similarly *versioning* is not mentioned, and a dokieli paper that is published online in a distributed manner (possibly even at multiple locations) and subject to collaborative editing would naturally have many different versions (e.g. pav:hasVersion), which could make it hard to cite which version has actually been reviewed, published, updated, etc.

It would also be important to relate these versions to each-other so you could know when to talk about the "same paper" in a more abstract sense, as with prov:specializationOf or SPAR FaBiO's use of FRBR relations between Work/Expression/Manifestation/Entry - this would particularly become relevant when a dokieli paper is accepted by a more traditional publisher which assign it's own DOIs and republishes the text (the Expression) in a slightly different Manifestation (probably breaking most of the RDFa links) at a different Entry (new URL).

One strong argument for semantic scholarly publication is that you have the possibility to break down the strong barrier between the article text and the traditional supplementary material, as you gain the possibility of a closer integration of data and visualizations from within the dokieli article - as the author has previously shown. One big challenge here is that the boundary of the article becomes blurred - some kind of aggregation of resources that constitute the article in the form of a research object or similar would help to mark out what is the scholarly unit that is actually being proposed for review, publication and citation - and also to help attribution-wise as you can break down what authors contributed where.

This system paper describes a flexible solution for concurrent editing of documents. Contrary to the leading centralised models of on-line editing the solution can be deployed in a decentralised way and cope with local and server-based scenarios. It is also entirely based on Web technologies and standards, making it a truly open platform everyone can easily build tools around.

The system is compared to things like Google Docs but my understanding is that concurrent real-time editing is not supported at the moment. This could be stated more clearly in the paper and some future plans for implementing it should be introduced. I'm especially curious how the authors will deal with editing conflicts in a decentralised and asynchronous deployment setting.

I took a good look under the hood of projects such as eLife Lens and Composer, and its newest imitator Manuscripts, but felt an underlying disappointment that none of these reached the needed level of accessibility, openness, and semantic design for true open and citizen science. Indeed, Manuscripts goes in the wrong direction. Diving into dokieli and SoLiD, i had at last the feeling of 'this is it': after decades of struggle to realise a usable semantic web platform, a breakthrough!

In turn, i disagree with the reviews of this paper calling for rejection (perhaps to a Kuhnian extent). The grounds given for rejection seem forgetful of the lessons learned from the 2+ decades of struggle to get here. The rough edges of the paper are appropriate given the nature of the medium addressed (the web). Dare i say it: with dokieli, a bit of the magic of first writing HTML in NCSA Mosaic in 1993 is coming back. To go along with some of the grounds given for rejection of this paper as a scientific work, would be to move backwards in terms of web science, HCI, and related (e.g. Tuftean) concerns.

In contrast, and in agreement with the authors, i feel salient issues to address fall under the category of progressively enabling dogfooding/usability for a widening population of researchers and social web users. At this early stage, an appropriate criticism seems to be that Dokieli is not yet a practically functional editor - for now, HTML5 + RDFa still need to be at least partly coded separately. Dokieli seems to be navigating into a new space that hybridises elements of static site generators, collaborative document editors, and faceted LD browsing, integrated with a decentralised social LDP (SoLiD). Is it an antipattern, then, to keep switching from dokieli into Warp to directly edit the HTML? Should text editors integrate into a dokieli / Linked Research workflow?

Dear authors,

What is the goal you would like to achieve most with dokieli?

How do you imagine it changing the world?

I’m asking because I think the work itself is exciting, but I’m unsure whether the channels you are using really help you achieve your goals. Is your goal really a publication at a workshop? Probably not… I’m quite sure you have higher ambitions, and that a publication is just a stepping stone toward something bigger. That said, are you sure a scientific publication is the most efficient stepping stone to reach that goal? I think not. And that’s probably the wall you’re hitting and will keep on hitting. You might be picking the wrong battles.

The anonymous reviewers ask for science—I can’t blame them, given the type of venue you are submitting to. I saw some comments pass by on Twitter that approached this focus of the reviewers in a humorous way. That’s totally fine, but know you can’t blame the reviewers for not accepting this work. Their reviews reflect fitness for purpose, not an absolute judgement of quality. I believe the work was indeed not fit for the particular purpose you chose it for.

Let me be clear and frank: what you are doing is not science. And it shouldn’t be. Not being science is not a quality judgment. Art is not science. A good novel is not science. A marvel of engineering is not necessarily science. That doesn’t make it any less useful. It just makes it unfit for scientific discourse.

On the engineering quality of the work, I have few comments. It’s a great solution with appropriate technologies. A few minor issues that come to mind:

• What exactly is dokieli, and what is an instance? Is it a document, platform, both?
• Why do you ship the application logic with the document (and what is the document)? Why not ship the document without code, and have an external UI? You hint at the possibility of this in the text, but this is not detailed, while I think it is an important point.
• Is the fact that RDFa does not support graphs a limitation, for instance, in the context of Nanopublications that typically involve graphs? What languages should a client of dokieli documents understand?
• I have some concerns about novelty, with regard to things such as identifiers, data storage, etc.

On the science aspect of the work, I have much more comments. I could not talk about this, because I know it is not how you like this contribution to be evaluated. But if you submit to a scientific venue, I feel I must comment on this.

My main issue: what is the problem? Despite the presence of a “Problem space” section, the entire document is structured as a “nice to have” rather than a problem that needs addressing. I’m not saying it isn’t nice, but “nice” doesn’t necessarily make a nice research paper. It starts with the abstract: you describe feature after feature, without arguing a need for them. You first need to convince me there is a problem before I can possibly start caring about it. The introduction continues this line of reasoning. Yes, Nelson is right that we imitate paper. So what? Why is paper so bad? Does paper lead to high costs? Does paper lead to a worse understanding? Missed opportunity doesn’t really result in disaster. The same holds for the evolvability argument. Why is evolvability important? What happens if we don’t do this? You believe that your solution “has the potential to both proliferate high quality machine-readable data, and enhance human-friendly knowledge sharing on the Web”. Why is that necessary? What is wrong with the ways in which we currently share knowledge on the Web?

Research starts from a problem. One presents a solution that addresses this problem, measure how well it addresses that problem in comparison to other solutions. This allows other people to improve upon one’s solution (and measure to what extent they do so). Exploring feasibility is not science. The Semantic Webber’s Guide to Evaluating Research Contributions (PDF, sorry) calls this “look, ma, no hands!”. It’s not research if you cannot fail. How do you know if you have been successful? How can you measure this? Why is your solution a good solution? What does my solution have to do to make it a better solution?

If you want my opinion, don’t try to be science. Don’t do it. You’d be fighting the wrong fight.

If you’re on a mission to make this way of publishing the default way, you have my support. But competing with research papers is not something you need to succeed with that mission. Trying to turn this into a research paper will weaken the contribution that you have. There’s nothing to measure here—and nothing to gain from that.

If you do want to turn this into a research paper, you’ll need to have a clear research question, a clear way of failing/succeeding, and a measurable way for others to improve upon what you build. But again, does this contribute to the eventual goal you aim to reach?

Finally, a wholly different kind of remark: the user experience. Please forgive my bluntness—I know that taste is subjective—but I found the aesthetics so poor that they really distracted my reading. A brownish background with a large sans-serif font does not provide me with a superior reading experience—which is the least I would expect from a project that aims to improve upon paper. There are various design flaws that make this text highly unpleasant to read, and that’s bad marketing. A couple of examples: I need to scroll 2 full screens until I get to the meat of the paper. Why do the author names and affiliations have to take an entire screen? There is a table of contents, but it is hidden behind a menu button (despite there already being a menu). The contrast is too low for pleasant reading. The drawings are too large and disconnected. The mobile experience is highly fragmented. I suggest to take a close look at the typography of sites like Medium, National Geographic, and Wired. Right now, you’re ignoring your most numerous audience: there will always be more reading than writing.

My advice for going forward would be to clearly identify your goal, and ask yourself what the best way is to reach it. Rejection of this work does not prove anything (right or wrong) if it is sent to the wrong place. Pick your battles. Fight only those you need to win. Don’t let the others distract you.

Best,

Ruben

This is an interesting and valuable tool. I believe there is a critical need for more tools like this, especially in Semantic Web and Web Science communities, that are averse to eat their own dog food.

However, I find it hard for this publication to be accepted on a regular venue, given its lack of hypothesis, methodology, etc. This is a problem in general when presenting new Semantic Web tools that are not considered "new scientific contributions", something I disagree completely.

I would like to see some linking from newly spawned instances to the original ones, maybe using PROV-O or other provenance vocabulary. I think linking the "ancestry" of an article is critical to enable a richer discussion. Apparently this can be done manually (via RDFa), but I don't think that is a good approach.

Figures 2 and 3 are confusing IMO

The link showing a growing number of articles shows an empty list.Fixed

I'd like to see a further discussion on what are the plans to test Dokieli in real environments. I can imagine three levels of testing: (1) use of Dokieli among SemWeb researchers as a minimal (2) IT/CS researchers and (3) scientists in general. I think the needs and abilities of each group will lead to changes in the approach Dokieli has taken. For example, it is not clear to me that non-SemWeb people are interested in adding RDFa annotations. If that is true (and this is just an hypothesis), how can we encourage them?

I think one of the most valuable aspects of Dokieli is the potential ability to include research data and other supplementary information, such as code. Although implicit, the authors would make a stronger case if they mentioned this particular use case, that I believe is critical in science. For example, it would be really interesting to be able to link papers to datasets, visualizations, iPython/Jupyter notebooks or other web-based data toolboxes.

Another missing discussion is about versioning. Even if it is not directly related, many of us do use git+latex or other combination of tools for collaborative editions that allow version control. Apparently this is not addressed by Dokieli.

Minor issues:

• CSS doesn't seem to work properly. I can't see any change when creating a H2 element.
• The authors forgot to mention Google Docs as a de facto standard collaborative tool on the Web that allows annotations.
• Menu on title appear too high and can't be used.
• There is no clear way to mark RDFa for the whole article.
• The RDFa option also forces me to copy and paste whole URIs. I can' imagine some service that provides autocomplete that could be useful for 90% of the times I want to add metadata this way.
• As any WYSIWYG editor, things can get really messy when inserting, deleting sentences in the middle of a paragraph. It would be nice to have a Markdown mode or something equivalent.

@csarven's #dokieli the future of #linkedresearch and #sempub scientific publishing? csarven.ca/dokieli @force11rescomm Beyond #pdf

SOLID data storage, Linked Data Notifications; several interesting things in @csarven's dokieli dokie.li

IMO researchers should move towards #linkeddata and dokieli: github.com/linkeddata/dok… . Kudos to @csarven and all who made it possible.

Since the page count greatly exceeds the maximum specified in this year's CfP, I must recommend to reject the paper.

This is unfortunate, since the dokieli system appears to be very interesting. Therefore, I would encourage the authors to rewrite the paper and resubmit it in a relevant conference.

At the same time, I would also strongly suggest to include a more comprehensive discussion of the advantages of this solution (e.g., which are the specific limitations that the system will address? How will it impact the users in term of usability? How does it improve on the state of the art?) and –crucially- an evaluation, since at the moment the paper presents no empirical evidence of the benefits of dokieli.

In general, most of the issues flagged by previous reviews at https://csarven.ca/dokieli still appear to be unresolved.

The dokieli tool is nice, well designed and implemented, and surely worth presenting to the community.

Though I do not think the research track is a good place for this kind of paper. It can be a very good poster instead. In fact it is actually a (detailed) technical report on the system, more than a research paper.

The requirements of the system are just listed in Section 2; a more detailed explanation of the process to derive them is needed. The paper should better frame the problem being addressed and the contribution.

The related works section is also just an overview of tools and does not identify the specific problems addressed by this work. The paper should also compare the proposed solution with the existing ones.

Section 4, and especially section 4.2, just lists some architectural choices, without really justifying them.

A lot of details could be removed from the implementation section; some screenshots could be removed as well.

The discussion is rather short, some end-user tests and usability evaluation is needed to demonstrate the robustness and applicability of this work.

The authors made available past reviews on this paper, most of which raised similar concerns, but they haven’t addressed these concerns yet.

A minor point: a lot of tools, libraries and technologies are mentioned in the paper but some references are missing (to websites, repositories, specifications).

The paper is 20 pages long. According to EKAW2016 call for papers:

"All submissions for research, in-use and position papers must be in English, and no longer than 15 pages. Papers that exceed this limit will be rejected without review."

The paper fails to fulfill the length requirement.

From a content point of view, the paper presents a very interesting approach/tool for document authoring, publication and interaction. I believe it has the potential to change the traditional way authoring and publishing is currently done. It truly shows how Semantic Web technologies can be used to create a collaborative and distributed apps.

Although the work itself is very interesting, the way it is presented, as a scientific publication, has many problems e.g. the problem addressed is not clearly defined, there is no methodology, and no evaluation.

This image will be updated from time to time to include the changes, e.g., social interactions, on this resource.

is there any plan to also address 'user adaptation' in dokieli? in particular I am thinking about multilingual content as well as other types of customization/personalization based on the context of user who reads a dokieli document.

This paper presents dokieli, a decentralized web-based authoring service.

The paper is very well-written and a pleasure to read. The service presented here is nice and useful, it defines a new way of publishing and interacting with content on the Web. There are many on-line resources that can be triggered to understand more about this service; indeed, the reader can access the github repository to gather more information and also the dokieli Website is a useful resource. Nevertheless, the paper itself does not provide many technical details and remains quite at a high level.

The contribution is well-suited for the conference and it should generate interesting discussions at the conference.

The authors present dokieli, a decentralised, browser-based authoring and annotation platform to create “living” documents that are interoperable and independent of dokieli. Dokieli follows standards and best practices, mainly HTML+RDFa for, Linked Data Platform for personal data storage, and Linked Data Notifications for updates. The presented platform has real potential in the context of academic/journalistic writings. The paper presents good arguments around the use of such a decentralised system to offer authors ownership and sovereignty over their data. The implementation is also sound. However, the paper lacks rigour from the perspective of comparing work against existing approaches. The authors do compare and contrast their system with other offerings; however, the research work and techniques used by other is not critically analysed (for example, there are almost no references in the main text. However, there is an excellent list of references at the end, was this intentional?).

The authors illustrate a decentralized framework for web-based authoring and annotation service.

The paper is really well-written, organized and motivated.

The main weakness is the nature of the proposal more fitting a demonstration context rather than an application/industrial scenario.

However the contribution will leverage a fruitful discussion during the conference.