Ice Year

Everyone has a corresponding star shining in this universe.

Arthur C. Clarke "2001: A Space Odyssey"

Introduction#

  Due to some controversial actions during Musk's transition of Twitter to 𝕏, a large number of users chose to migrate from Twitter to platforms like Mastodon, Threads, etc. The term Fediverse has gradually entered the public eye, and even Threads claims it will support the ActivityPub protocol in the future to achieve interconnectivity and join the Fediverse (which has even sparked a series of discussions within the Fediverse community).

  On February 6, 2024, Bluesky announced the end of its public beta that began in 2021, allowing users to register without an invitation code. ¹ As a rising star in the Fediverse created by a team of Twitter co-founders, Bluesky did not adopt the currently popular ActivityPub protocol in the Fediverse, but instead developed its own AT Protocol (Authenticated Transfer Protocol).

  Having followed the Fediverse for over two and a half years, I browsed the relevant content of the AT Protocol out of curiosity, and in the spirit of sharing, I am documenting my thoughts and understanding of the AT Protocol in this article, which is neither a thesis nor a report nor a blog post. Due to my limited level, this is for reference only, and I welcome corrections and discussions.

Initial Impressions and Comparisons#

  This analogy may not be very appropriate, but if we compare the development of blockchain, I believe that protocols represented by OStatus are similar to Adam Back's hashcash, ActivityPub is akin to Bitcoin, while AT Protocol feels somewhat like Ethereum and EOS.

  If you have previously interacted with instances in the Fediverse based on the ActivityPub protocol, you may be aware of the connectivity flaws between instances: it is impossible to migrate to a new domain when the original domain has ceased to function, and interactions with other instances can only display interactions from instances that are already federated with your own; if you have ever operated such instances, you may also have experienced the financial burden of the Fediverse (which has successfully made IDC providers like Hetzner the hosting choice for over half of the instances in the Fediverse, humorously dubbed the capital of the Fediverse), as well as the exhaustion of having to manage both technology and administration. Of course, this does not prevent ActivityPub from being one of the very excellent protocols in the Fediverse, but these bugs always lead one to ponder whether there are ways to improve.

  In 2023, Nostr, represented by blockchain-based Web3 social media, gradually "went viral" due to promotion led by Edward Snowden, leading to a surge of "mysterious codes" on Twitter. Nostr introduced blockchain into social platforms, adding a layer of value (or incentive layer) and achieving interconnectivity through P2P protocols like DHT. However, due to Nostr's design philosophy, it lacks an inherent mechanism for information filtering, and the current lack of cross-chain mechanisms, along with the high costs and risks of operating chains or nodes, has resulted in users being primarily concentrated on the main chain of Nostr. The "mysterious codes" of Nostr also reveal flaws in this social approach: a lack of readability. While computers can easily interpret hashes, the vast majority of humans cannot memorize large amounts of randomly generated hash addresses. Although other blockchain-based Web3 social projects like Farcaster and Status I have previously experienced attempt to map using human-friendly methods like ENS, the high costs of ENS hinder user participation.

  Moreover, whether it is ActivityPub or implementations similar to Nostr and Farcaster, they all adopt absolute timelines and follow push notifications. While this avoids the "information cocoon" caused by a single platform's algorithm, it leads to another problem. Since users have limited attention, ActivityPub prioritizes (or even exclusively pushes) information from its own instance and federated instances, requiring users to filter information based on time order, thus shifting the filtering cost onto the users. Similarly, Nostr employs a time-based mechanism, but the centralization of the main chain leads to information overflow that exceeds any large instance using the ActivityPub protocol, evolving from an "information cocoon" into an "information explosion." The design of the ActivityPub protocol means that users can only take a one-size-fits-all approach to blocking information or wait for instance administrators to delete or block information, without considering the potential to reduce push demands; Nostr is even worse, as the vast amount of information makes it nearly impossible for users to filter while exploring the timeline, and both client-based filtering and manual review become exceptionally difficult.

Architecture of the AT Protocol#

  The brilliance of the AT Protocol design lies in its separation of data storage, pushing and discovery, filtering, and the user interface, achieving openness, friendliness, disaster recovery, ease of migration, and scalability.

  As Bluesky elaborated in its paper: the design of the AT Protocol references today's internet, ² where websites should not all bear the responsibility of mutual discovery and federating all websites on the internet, nor should they all bear the burden of filtering and screening for the entire internet. It can be said that the World Wide Web is currently the most successful decentralized autonomous organization, and referencing it is undoubtedly very wise, greatly reducing the cognitive load and workload. The internet has hardware, software, and standards, and the design of the AT Protocol can also be said to have hardware, software, and standards. However, just as the internet has blurred the boundaries between software and hardware through virtualization and container technology, it can be anticipated that the software and hardware of the AT Protocol will not be limited to the original standard implementations, but may also see integration and expansion in the future.

Hardware#

  Assuming that the hardware in the AT Protocol refers to traditional hardware in the internet, then in the official standard implementation of the AT Protocol, Personal Data Server (PDS), Relay, and App View ³ can be considered the hardware of the AT Protocol.

Personal Data Server (PDS)#

  The Personal Data Server is, as the name suggests, a place to store user data. In previous implementations of ActivityPub, this relied on various instances, but in the design of the AT Protocol, the Personal Data Server is equivalent to a server on the World Wide Web, only needing to implement the storage of relevant data, significantly reducing performance requirements and costs, and can even be set up on home NAS and Raspberry Pi. This is expected to make the decentralized network built on the AT Protocol more robust due to the addition of more nodes. Additionally, just as virtualization technology allows a single independent server to host multiple virtual servers, the Personal Data Server can also aggregate operations to provide services. For example, the official operation of Bluesky is currently the largest aggregation of Personal Data Servers, similar to IDC in cloud computing.

Relay#

  The Relay is akin to a gateway and router in the internet, serving as an intermediary connecting Personal Data Servers. The first component of the Relay is the Relay Server, which is responsible for fetching push data from known Personal Data Servers, verifying the data, discarding data that fails verification, and initially filtering out high-frequency spam. The Relay aggregates this data through the Relay Server to create a Firehose ³, which is then received by the App View ³. The benefit of this approach is that it significantly reduces the number of requests to both Personal Data Servers and App Views compared to subscribing to multiple Personal Data Servers separately, and it can be verified through a timestamp-based Merkle tree. Similar to how there are few operators responsible for operating lines and gateways in the internet, operating a Relay is the most costly part among all nodes in the AT Protocol, and its expected quantity is relatively small.

App View#

  The App View is the component most apparent to users, comparable to terminals accessing the internet, or more vividly, APPs. The App View receives the Firehose from the Relay, processes the corresponding records according to the Lexicon, corresponds to counting the likes of each post in Bluesky, organizes the reply threads of posts, maintains each user's collection of followers, and constructs a timeline containing posts from followed users, while also parsing media file addresses in the information and delivering the parsed files (with necessary processing, such as compression) to users. Just as nearly all connected applications fundamentally communicate based on TCP or UDP, the App View also communicates based on the AT Protocol. However, TCP and UDP only provide communication protocols, while the input, parsing, and output of the transmitted content rely on APPs. Based on the AT Protocol, many applications similar to the World Wide Web ecosystem can be developed. Currently, the largest application is the Bluesky microblogging platform, but it can be anticipated that in the future, many App Views similar to those in the existing Fediverse will be developed.

Software#

  Similarly, defining the software in the AT Protocol as the traditional software part of the internet, in the official standard implementation of the AT Protocol, Feed Generators, and Labelers ³ can be considered the software of the AT Protocol.

Feed Generator#

  The directory and search of websites rely more on search engines and their algorithms. Correspondingly, the Feed Generator in the AT Protocol serves the role of indexing and selectively pushing information. Just as users can freely choose to write their own search algorithms and build search engines, they can also freely choose to build their own Feed Generators. Bluesky also provides a convenient starter kit on Github. ⁴ Moreover, similar to SearXNG, which allows users to select and aggregate information from search engines, users can also choose and aggregate Feed Generators in the App View to obtain and explore information, simply by clicking to add, just like downloading an APP from an app store. In other words, a large number of Feed Generators forming an algorithm market itself is the vision of the AT Protocol.

Labelers#

  For filtering and screening website content, it relies more on ad blockers and fraud protection, or firewalls. The Labelers in the AT Protocol serve a similar function. Just as users can subscribe to and stack different ad blocking rules, they can also use different labeling nodes to tag data, achieving more refined filtering and control, catering to the diverse needs of different users while effectively separating operational and management tasks.

Standards#

  Continuing with the previous definitions, the standards in the AT Protocol can be defined as the traditional standards part of the internet. In the official standard implementation of the AT Protocol, DIDs (Decentralized Identifiers), Handles, Repositories, and Lexicons ³ can be considered the standards of the AT Protocol.

  Behind the standards is the data model of the AT Protocol. The AT Protocol applies various resource identification specifications, currently including DIDs, CIDs (Content IDs), NSIDs (Namespaced Identifiers), and rkeys (Record Keys). ⁵

DID (Decentralized Identifiers)#

  Due to the brilliant design of the identity identification system in the AT Protocol, it deserves to be singled out (not really)

  Well, it's just because the length is considerable, but the design is indeed clever.

  DID superficially resembles an account system on the internet, but I believe it is more akin to the implementation of domain names on the internet, and it largely references the internet's certificate issuance mechanism. To be precise, DID is not an original invention of the AT Protocol; the true inventor is W3C, and there are currently hundreds of DID methods (evidencing humanity's penchant for tinkering). ⁶ After discussion, the developers of the AT Protocol decided to adopt only the did:web and did:plc methods to better avoid overtime meet requirements.

  The cleverness of DID lies in that it is not just a simple combination of characters, but can be resolved into a JSON array called a DID Document ³ through a mechanism similar to DNS records on the internet (essentially still a character combination). The DID Document in the AT Protocol follows the Merkle tree specification to ensure immutability and verifiability, so the Merkle tree of the DID Document will gradually grow over time and with operations.

  However, because DID often adopts a hash-based compression mechanism, it frequently appears as a combination of random characters, which is not user-friendly. To address this, the developers of the AT Protocol adopted the Handle approach to solve this issue, establishing a mapping between domain names on the internet and DID through a human-friendly domain name mechanism, using TXT records or the /.well-known/ path via HTTPS (familiar to those who know about free SSL like Let's Encrypt). Therefore, in Bluesky, a username like @xxx.com (which indirectly resolves the blue verification issue, as long as example.com is the official website of a person or organization, it essentially obtains verification, shifting the verification responsibility to ICANN).

  The implementation of the did:web method is similar to the Handle mechanism, where the user's DID is the domain name, and the mechanism for resolving the DID to return a JSON array is done by querying the specific TXT record corresponding to the domain name, with the return value being the DID Document (in this case, we can remove "similar"; it is indeed DNS resolution/doge), or by accessing the /.well-known/ path via HTTPS to obtain the DID Document. This approach combines Handle and DID, achieving user-friendliness. However, the downside is that the domain name serving as both DID and Handle is immutable due to the immutability of DID, making it susceptible to real-world events like domain changes. Additionally, with each growth of the Merkle tree, manual updates to the TXT records or /.well-known/ path are required, which can be inefficient in large-scale applications and may lead to delays in updates or potential hijacking due to varying network conditions.

  The did:plc method was created to address these pain points. It is called did:plc because the developers originally intended it as a "placeholder" (but clearly it is no longer), and there is no need to worry about the future of did:plc, as the AT Protocol provides backward compatibility in its corresponding implementations of DID to avoid potential obsolescence caused by the addition of new DID methods. The did:plc method implements encryption and compression of the DID Document using the secp256k1 and NIST P-256 algorithms from did:key, possessing native encryption characteristics.

  The request process for did:plc is as follows: send a request to the PLC Server ³, which checks if it has already been registered. If registered, it returns a status code and, if necessary, the DID Document. If it is an update operation for the DID, the PLC Server will make a determination through a verification mechanism. The PLC Server also decides which version to retain in the event of a fork in the DID. The PLC Server offers various verification mechanisms to choose from, including using the rotating key pairs provided by ddi:plc for verification, or utilizing a hosted signing key pair by the PLC Server and employing traditional email-password mechanisms for verification (for example, the official PLC Server of Bluesky), achieving a balance of convenience and security.

  The application of DID makes data migration within the AT Protocol exceptionally easy; users only need to update their DID Merkle tree, adding records of changes to new Personal Data Servers, and transferring backup media files from the original Personal Data Server to the new one. Even if the original Personal Data Server has completely vanished, the original social network can still be retained. Additionally, through the combination of did:plc and extended Oauth 2.1, it is even possible to log in to applications compatible with the corresponding Oauth standards using the AT Protocol. ⁷

Lexicon#

  If you have played in the Fediverse based on the ActivityPub protocol, you should have some impression of the differences between various projects: for example, Mastodon only allows likes for a post, while Misskey can have multiple reactions. Both Mastodon and Misskey use ActivityPub as the protocol for interconnectivity, but to enable Mastodon to support Misskey's reactions, modifications to the entire Mastodon project would be necessary. This design is not very elegant, while the design of the Lexicon in the AT Protocol elegantly resolves this issue.

  The architecture of Lexicon references JSON and OpenAPI. Personally, I feel that NSID serves a role similar to Android package names, while rkey is somewhat akin to individual Activities. The namespace mechanism of NSID makes Lexicon more extensible, allowing projects developed based on the AT Protocol to focus solely on designing the Lexicon and corresponding elements in the App View, making interconnectivity and interoperability in the Fediverse more convenient than ever.

Repository#

  If you are familiar with IPFS, seeing CID should evoke a sense of familiarity. You are correct; the Repository in the AT Protocol follows the IPLD specification and implements version control through the Merkle tree mechanism of Git. The advantages of content addressing in the IPLD specification, avoiding duplicate data, and the signature verification and version control of Git and Merkle trees will not be reiterated here.

Conclusion#

  It can be said that the development of the AT Protocol stands on the shoulders of countless giants, from the ubiquitous mapping of internet elements to its claims of not being blockchain ⁸, yet its body honestly embraces the extensive applications of Merkle trees, and the unprecedented customizability of pushes reflects the shadow of RSS, as well as the application of Git in the data model. The AT Protocol learns from the pitfalls of Twitter, ActivityPub, Nostr, and others, continuously evolving and rising to become the most promising form of Web3 social protocol.

  Currently, the application of the AT Protocol still exhibits a relatively centralized phenomenon, with Bluesky often being the only choice in many cases. However, if it continues to develop according to Bluesky's declarations, the AT Protocol will become increasingly open, and it is foreseeable that its excellent features will attract numerous developers, promising a bright future for its ecosystem. There is a peculiar feeling that 𝕏 resembles the Netscape company of yesteryear, and from it, Bluesky hopes to become an open organization like Mozilla, promoting technology towards a more open, transparent, fair, and interconnected direction.

Original article link: https://blog.iceyear.eu.org/atprotocol-study