Since 2010, when Frank kicked off a private cloud storage project as part of the KDE community, the Nextcloud code base has advanced significantly as it evolved from a basic file-sharing tool to a full collaboration platform rolling out to hundreds of thousands of new digital work places every month. Today, we announce a significant rewrite of Nextcloud’s advanced file access layer, introducing the ADA engine. Written in PHP, Go and Rust, the new core will bring the already impressive scalability of Nextcloud to a new level.
ADA stands for Accelerated Direct Access but is also a homage to Ada Lovelace.
Ada Lovelace was an English mathematician and writer chiefly known for work on Charles Babbage’s proposed mechanical general-purpose computer, the analytical engine. She was the first to recognize that the machine had application beyond pure calculation. Lovelace is often considered the first computer programmer.
Source: Wikipedia
Scaling complex collaboration
Over the last 10 years, Nextcloud has become the most advanced, truly sovereign collaboration platform on the market. Its File Sync and Share application, Nextcloud Files, is unmatched in its rich features:
- The most flexible sharing solution on the market, supporting:
- Any mix of individual, group, and team folder shares, as well as secure File Drop
- An unlimited number of individually configurable and named public shares
- Advanced permissions, password & expiration settings, video verification, and even full Access Control List permissions
- The most extensive security capabilities, including:
- Flexible server-side as well as end-to-end encryption
- Advanced, tags and rules-based file access control, including automatic sensitive-file detection based on content and metadata
- AI-powered suspicious login detection, brute force detection, rate limiting, audit logging, and many other built-in security mechanisms
- Smart file locking, either manual or automated, occurs when users open a document locally or on their desktop or mobile device using the apps. Including the ability to choose to open any document directly from the browser in a desktop application, with automatic re-upload once the user is done editing the file locally, and file locking to avoid conflicts.
- The most advanced storage abstraction, supporting any number of file storage solutions of nearly any type, from any POSIX file system, S3, and IBM storage tech to FTP, WebDAV, Samba, NFS, and Microsoft SharePoint.
- Advanced clients for MacOS, Windows, Linux, iOS, and Android, which all support syncing documents for offline availability and give access to all features. Push notifications sync data instantly and notify users of changes.
- Extremely flexible deployment, from bare-metal or VM-based clustering to advanced, cloud-native managed container platforms like Kubernetes, with a wide range of database options.
This is just an incomplete list, but on top of it, Nextcloud offers a wide range of unique collaboration features. For file management, this ranges from adding rich context to folders like checklists, documentation, and links, commenting on files (extensible to a full chat with video call), to support for building easy approval workflows.
The platform itself also evolved, with many applications built on top that are industry-leading in their own right, such as Nextcloud Talk, the most extensive sovereign Microsoft Teams replacement on the market, as well as Nextcloud Office, Groupware, and other components.
Besides applications built by the core community together with the engineers at Nextcloud GmbH, a wider community of countless other independent developers and organizations, both public and private, have developed applications on top of the open platform.
These all deeply integrate with each other and the Nextcloud Files application. Think of the possibility of sharing a file directly to a task so it can be attached, or sending a file into a Talk conversation to present it during a call or work on it in real time with others.
Users can find documents related to a conversation, see them attached to a calendar item or task, link them in a table, or store incoming documents from emails into Nextcloud Files. This is combined with rich, real-time business workflow automation, which can act on this data.
Last but not least, all data across the platform can be utilized through the Nextcloud Assistant to assist users in answering questions about themselves or act as an agent, sending messages, analyzing data, generating documents, and much more.
As we earlier argued in an opinion piece, this integrated collaboration is clearly the direction the industry has taken since 2015, and it is the type of solution that Microsoft, Google, and others have built. It is therefore a hard requirement to ensure relevance and provide true digital sovereignty for our community.
This means Nextcloud’s core architectural challenge is managing complex, shared state and ensuring tight cross-feature consistency while ensuring security constraints.
In such systems, features are not independent: A file shared in a chat, referenced in a project, and edited collaboratively must always reflect the same data and permissions. Similarly, a chat & video conference room is real-time interconnected with tasks, documents, and meeting appointments.
A new architecture for collaboration in Nextcloud: the ADA engine
To take on the challenge to scale such a deeply interconnected system while ensuring strict and consistent data access controls and security, the Nextcloud team has rewritten sections of the core file handling and abstraction logic with components in PHP, Go, and Rust.
Dubbed ADA, which stands for Accelerated Direct Access, the new engine is designed to pre-calculate and cache the data access and permissions, offering direct file access and actively pushing data to the clients, ensuring responsive access when users navigate the system. The new engine offers:
- Direct data access through pre-calculating access rights and other data
- Direct file downloads bypassing the server (on S3)
- Direct data push to the clients to reduce traffic
Through the new ADA engine, Nextcloud Hub is able to continue to expand the most advanced sovereign collaboration platform with new, real-time integrated collaboration capabilities without having to compromise on performance. The choices Nextcloud has made in the technology space have been a key factor in achieving its position as a clear market leader in the sovereign collaboration space, and the ADA engine will further solidify this unique advantage.
Architecture updates for Nextcloud Hub 26 Winter
Let’s dive a little deeper into the changes that come with the upcoming release of Nextcloud Hub 26 Winter. To accelerate data access across the platform, the team worked in three areas:
- The File Cache: Re-organizing the core database and IDs in a way that splits tables and creates a base for more advanced sharding support in the future
- Lean data access: Replacing the mount cache with an authoritative table that shifts work from read to write actions and enables lean, direct data access in the file system
- Direct downloads: Ensuring support for direct downloads of files and previews on large instances using S3
Besides these changes to the core, the High Performance Backends for both Nextcloud Files and Talk, written in Rust and Go, were improved to push more data directly to clients:
- Version 2.0 of the High Performance Backend for Nextcloud Files, a component written in Rust, spreads the load over an interval to different connected users and sends a list of files impacted by changes to the listening desktop clients to reduce the need for active checks.
- The update to the High Performance Backend for Nextcloud Talk, largely written in Go, now relays chat messages and the participants in Talk rooms through the HPB, significantly reducing server load for large chat rooms.
Let’s have a look at the technical details!
The File Cache: splitting and sharding
Today, the File Cache is home to data on various types of files (and directories), including the users’ files, previews, avatars, and files of apps themselves. The File Cache is often jokingly called the worst-named part of Nextcloud.
It is anything but a cache, as it serves as a unification/abstraction layer to store important metadata for files. There are two reasons for doing this.
First, not all supported storage technologies support all the metadata Nextcloud uses.
Second, database access is faster than going to storage, especially when the required data is complex and interconnected.
Keeping the data in one place rather than replicating between a cache and real storage also avoids synchronization and cache invalidation challenges.
With the Nextcloud Hub 26 Winter release, we are bringing 3 changes to the File Cache:
- Halving the table size
- Sharding groundwork and Snowflakes
- Streaming database lists
We’ll explain these in more detail.
1. Halving the table size
The File Cache’s abstraction is a bit overkill for some file types, while at the same time, it is often the single largest table in the database.
So in Nextcloud Hub 26 Winter, previews have been split out of the File Cache. As each file can have multiple previews, this can significantly reduce the size of the File Cache table.
Previews have their own unique peculiarities, which the team could adapt the new table to. In particular, it makes sense to expire previews when not accessed for a long time, a feature that now comes to Hub 26 Winter.
In our production-like test environment, this change resulted in a 56% reduction in the size of the File Cache table. Systems without previews will, of course, see little benefit, while servers that use previews can easily see a 50-60% reduction.
2. Sharding groundwork and Snowflakes
Besides halving the size of the File Cache table, the team has worked on a different type of database splitting: sharding. Sharding means dividing a table from a database by a certain property, like user or file identification number, across different nodes. Each part would be a “shard”.
One requirement for this is a suitable, unique identification number or ID across all the shards. This is where Snowflake comes in.
Snowflake IDs, a method to generate IDs developed at a site formerly named Twitter, are inherently more suitable for decentralized or clustered setups.
Right now, when the server generates a preview, the application server (Nextcloud node in a cluster) will ask the database to create a file ID for it by inserting partial data, then store the preview and update the previously inserted partial data.
With Snowflake, the application server simply follows a defined set of rules to create an ID and stores it in the database. This means less round trips and reduces wait times when the database is busy.
We will use the Snowflake ID in a few places. In the future, this will include the file_ID. This will mean file ID’s wont be in the familiar 100 range when you create a new Nextcloud server, but they will always be 64-bit. For developers, Snowflake IDs documentation for Nextcloud can be found here.
Of course, this is a work in progress. The basic work is done, and various components like the preview provider and external sharing provider have been updated. Specific setups at some of our larger customers, guided by a close collaboration with our team, will already begin to benefit from the sharding improvements in this release. These benefits will expand to larger instances over time.
3. Streaming database lists
There is one more aspect to improve scalability on very large instances, which is the handling of large lists. Think of making a change to all users in a system, like updating their email address.
On a server with 500 users, the code might ask the database for a list of all 500 users and then proceed to run the operation, figuring out what their new email address should be and applying the change. But when the server is operating with 4 million users, just retrieving and temporarily storing that list alone might require a significant amount of network and memory resources.
On such servers, it might be wiser to retrieve one or a few users at once, apply the operation, and fetch the next bunch.
This is only relevant in specific places where large data sets need to be handled, but those are not that hard to find on large servers. In those places, new API’s are devised that expose a way to get query results as a stream of smaller batches rather than all at once (generator pattern).
This won’t really affect smaller instances, but avoid out-of-memory or other performance challenges on the bigger servers.
Lean data access
The main goal of lean data access is to reduce the amount of work needed to process a request. This release makes big changes in two major areas: the mount points cache and the file system setup.
1. Authoritative mount points
The first change we did was to replace a cache of mount points (this includes shares and external storage) per user with a table that will always be kept up to date.
This means we no longer need to build the cache when a user requests data. This could happen more frequently than intended when the cache was created, as there is a fair bit of effort put in to ensure whenever there might be an inconsistency, the cache is recreated. So, going forward, when a user requests data, it will be there.
On our test instance, loading a folder went from 1.9 to 1.3 seconds, reducing the time to load by some 30%. As this was a fairly normal folder, this would be a typical case. But folders with many shares, or users who have many shares in general, can see far larger improvements.
Moreover, this change impacts a great deal of requests and will also reduce the variation in request duration. Last but not least, it benefits large, clustered setups in particular.
As maintaining the table is no longer done when a user requests information, it has to be done when shares and storage are modified. This can slow down operations like creating a share or new storage. But these operations happen far less often, and can be optimized better. The total reduction in load is thus helpful, but the real impact will be increased responsiveness for users.
Right now, the groundwork for the new table is done, and it is implemented in the External file storage app and the sharing app. Other apps, like Nextcloud Teamfolders, Collectives, Deck, Talk, and Circles, are in the process of implementing the new mechanism.
2. Lean file system setup
The second improvement similarly required changes to the file access abstraction. Retrieving a file means loading data from all file providers as well as all mounts (shares) a user has access to. This means processing quite some data while the user might only be looking to retrieve metadata on a single, specific file.
Thanks to the new mount point setup, the mechanism to ask for file providers now allows a request for a single file storage provider. Providers, in turn, were updated to allow asking for a specific path of a file. This made the storage providers more complicated, in a classical trade-off between complexity and performance.
The impact of that trade-off is not insignificant, however. In a production-like test, retrieving the contents of a shared folder went from 1.39 seconds down to .44 seconds, shaving off nearly 2/3rds of the time.
Combined, the authoritative mount points and the more efficient file system access mechanism will greatly improve the responsiveness to users on larger systems, exponentially so the more data and shares these users have access to.
Direct downloads
The last major area of improvement has been to make it possible for clients to directly grab files from S3 buckets. While currently limited to S3-compatible storage solutions, others that can offer a token-protected direct download could be supported as well.
Most large Nextcloud instances use S3-compatible storage technologies. When a client in such a setup requests a file, it is transferred from storage to the application server, which then makes it available to the client. With direct download, a client would instead receive a direct download link from the application server, which it could then use to grab the file directly
