Building a decentralized collaborative text editor using MQTT and CRDTs
Intro
People across the globe work in distributed workflows use (a)synchronous collaborative tools like Google Docs or similar. Usually, these applications running in a browser communicate with backend services where the actual content is stored. The demo we introduce in this article does not require a backend service to store documents but rather uses an MQTT broker to enable very lightweight communication. Moreover, the problems become increasingly difficult when the internet connection is intermittent, e.g., working from an airplane or in regions with low internet coverage. Changes must eventually merge into a conflict-free document.
In the article, we introduce the concept behind CRDTs, Automerge, and how to apply MQTT to build a distributed workflow. The source of the editor is available at GitHub and a demo runs at
https://hivemq.github.io/distributed-mqtt-editor/distributed-mqtt-editor-demo
The example described in this blog post illustrates the use case of MQTT for distributed user workflows, and highlights its simplicity and lightweight protocol.
CRDTs and Automerge
In academia, researchers proposed numerous data structures to tackle that problem range from completeness to correctness. One important data structure is the Conflict-free Replicated Data Type (CRDT) generalizes the concept of editing replicas independently, which eventually converges without conflicts.
Software developers apply CRDTs to implement collaborative tool features such as text editing. Another prominent example is distributed data structure in the in-memory database Redis.
A framework that implements CRDTs is Automerge , which is available for programming languages such as Rust, JavaScript, Go, and Swift. As an alternative to Automerge, you can also use yjs. To illustrate the idea, the following JavaScript code merges two text documents (replicas) into one, assuming the application runs in a browser. The replicas may be edited by independent authors who eventually wanted to have a single document:
function merge(othersText: string): string {
let newDoc = Automerge.change(currentDoc, doc => {
doc.text = new Automerge.Text(othersText)
});
currentDoc = newDoc;
}To transmit a replica to another author, the current state or the edited changes must be replicated. Automerge provides the following save-function to export the document as an UInt8Array.
function getLatestState(): Document {
return Automerge.save(currentDoc);
}The getLatestState call is transmitted to all other editors, so no further dedicated backend is required to handle those changes. Once a message reaches an author’s browser, the function merge is called to apply changes from all other authors.
Text Editor with CRDTs over MQTT
The concept behind CRDTs does not specify anything related to a network transport protocol. Instead, it requires synchronization to eventually happen to create a helpful use case as illustrated above. MQTT is a lightweight communication protocol that efficiently transports necessary information about the updated document.
Combining these two technologies opens up the door for the following use case. Automerge provides different chunks between the previous and the latest document. With that, we publish the lightweight changes as an Uint8Array via MQTT and merge them in all other editors to fulfill synchronization of the distributed state.
With this process, the authors can independently update their document (replica). Automerge is then used to keep track of changes and to merge other replicas conflict-free. MQTT transports the changes to all other authors. There is no central backend service to manage the changes. Each author has its own replica standalone in the browser, but all converge to the same content.
The diagram below illustrates the data flow between two authors Alice and Bob.
Alice and Bob are asynchronously updating a document
Alice and Bob are typing “MQTT”, where Alice continues with typing “is great” and Bob “:-)”. Both updates are replicated using MQTT to Alice and Bob, respectively, which converges to the document “MQTT :-) is great”.
The scene is illustrated in the following diagram:
Alice and Bob working in a browser
Topic Structure
We use the following MQTT topics to transmit updates to all authors.
Consider the following JavaScript code:
const topicPrefix = process.NODE_ENV.MQTT_TOPIC_PREFIX
export function documentTopic(docId: string): string {
return `${topicPrefix}/${docId}`
}
export function documentUserTopic(docId: string, senderId: string): string {
return `${topicPrefix}/${docId}/${senderId}`
}
export function documentUserCursor(docId: string, senderId: string): string {
return `${topicPrefix}/${docId}/${senderId}/cursor`
}The variable topicPrefix is set by a configuration variable and can be freely chosen complying the MQTT topic names. The docId is a user-defined identifier. In practice, that would mean that Alice and Bob chose the same id to work on the same document. Each user gets a randomized senderId to identify the author.
Each document update is transmitted via the MQTT topic ${topicPrefix}/${docId}/${senderId} whereas each author of the same document is subscribing to ${topicPrefix}/${docId}/+ to get updates from all other authors.
Any cursor position change is transmitted via the MQTT topic ${topicPrefix}/${docId}/${senderId}/cursor and each author is subscribing to all cursor changes via subscribing to the MQTT topic ${topicPrefix}/${docId}/+/cursor.
Implementation
For our demo use case, we implemented a React application that uses the Quill as editor component. We use the JavaScript library MQTT.js as MQTT client library. Checkout our GitHub repository for this demo get your hands on. If you want to continue with the use case, consider using our HiveMQ Cloud Free offering to get a secure and private broker in no time.
Andrej Schujkow
Andrej is Senior Frontend Engineer at HiveMQ. He loves to create complex web applications in a simple fashion, enable developers and career changers, reverse engineer legacy features and absorb the latest frontend tech.
Stefan Frehse
Stefan Frehse is Senior Engineering Manager at HiveMQ. He earned a Ph.D. in Computer Science from the University of Bremen and has worked in software engineering and in c-level management positions for 10 years. He has written many academic papers and spoken on topics including formal verification of fault tolerant systems, debugging and synthesis of reversible logic.
