Blockchain data indexing platforms are one of the most consequential, yet largely unknown innovations in the Web3 industry, helping to free up developers’ resources and dramatically improve the user experience in decentralized applications. Yet the vast majority of Web3 users don’t have the first clue about what a Web3 data indexing platform is, or does, or why it’s so important.
It might come as a surprise to learn that data indexing is what makes the Web2 world go around. Indexing facilitates the bulk of people’s online interactions, enabling YouTube to serve them videos in interesting genres they like, ensuring search engines deliver results in their language, and making sure that your Facebook feed is populated by things likely to interest you, such as what your friends are saying.
But nobody thinks about indexing, because it’s a back-end thing that just makes everything work. So long as the internet does work, why would anyone care about how it works?
What Is Blockchain Data Indexing?
To understand the importance of indexing, it might be helpful to think back to the early days of Web1, when the only way to find something online was to manually type the IP address of the website you want to visit into the URL bar. That changed when Google came along and indexed the web for the first time with its now-eponymous search engine
Now, indexing is going to have a similar impact in Web3 and the tangled web of blockchains that supports it.
For developers, indexing is a godsend. Imagine you’re building a dApp and you need real-time access to blockchain data, user’s interactions and smart contract events. Finding all of this information on the blockchain, which successively stores data, is an extremely painstaking task.
The blockchain acts as a distributed database, serving as the backend of every dApp. It provides all of the data a smart contract needs to function, but the challenge lies in accessing this information. While searching for data within a specific block or an account is fairly simple, it quickly becomes much more complicated when developers need to aggregate data from multiple queries of single blocks
The problem is the sequential nature of blockchain, where data is stored one block at a time. As the blockchain adds more and more data, it becomes longer, leaving the data within it scattered all over the place. It’s incredibly disorganized for anyone trying to search through that data and make sense of it, and so it takes a long time to retrieve that information. Yet the most advanced dApps of today require extremely large amounts of blockchain data processing logic and often need to do this for multiple chains at once.
This is where blockchain indexers such as The Graph and SQD play such an important role, helping to streamline the way on-chain information is categorized and indexed. They can be thought of as specialist tools for developers, and they vastly simplify the process of building dApps. Just like the index of a book, a blockchain indexer makes it quick and easy for the user – in this case a smart contract or dApps – to find the information they’re looking for. Blockchain indexers help to make the process of querying, retrieving and utilizing on-cain data more efficient.
For developers, the beauty of a blockchain indexer is that they only have to define the data type and the relationships between them and specify how that information should be stored. The blockchain index will then create a customized GraphQL endpoint that presents the necessary information to their dApp. This solves all of the blockchain data retrieval and querying headaches for developers, leaving them with more time to focus on the actual functionality of their dApps – in other words, making them better.
Blockchain Indexing Benefits
Blockchain data indexers abstract away much of the complexity, they’re optimized for latency so they can access information in real-time, as it’s added to the blockchain, and they can retrieve historic data rapidly too.
The benefits for developers are myriad, helping to speed up dApp development lifecycles, increase the responsiveness of dApps, boost their reliability and provide a smooth user experience.
For one thing, blockchain indexers are developer-friendly, abstracting away the complexity of coding logic that was previously required to query and retrieve blockchain data. They automate much of the process of indexing blockchain data, using templates and quickstart guides to help developers get started. Some of the newer indexers support familiar programming languages, such as TypeScript and JavaScript, meaning they’re easier for developers to get to grips with.
Blockchain indexers support a high level of customization too, enabling developers to quickly and easily index application-specific data such as custom events from smart contract functions. At the same time, many also offer multi-chain support, meaning they provide developers with the ability to obtain and aggregate data from multiple chains and present it in a unified database table. This means dApps can quickly present unified information to users, without the developers needing to worry about using multiple APIs or subgraphs to access and manage the data from each chain.
The Evolution Of Blockchain Indexers
The very first blockchain indexer to make its mark on the Web3 industry was The Graph. As the first decentralized service of its kind, it gave developers free access to indexing capabilities, meaning they no longer had to pay a centralized service provider.
The Graph was once known as the “Google of Web3”, and provided an open data marketplace powered by the GRT token that any dApp could tap into. The Graph’s network unites numerous network participants, including data consumers, indexers who query data on their behalf, and the curators who manage the subgraphs.
Subgraphs were The Graph’s key innovation. Essentially, they’re a kind of schema that defines how data from a blockchain should be retrieved, indexed and structured, making it instantly queryable. To date, there are more than 1,700 subgraphs available via The Graph.
To ensure the accuracy of data queries, The Graph established a dispute system where anyone can challenge the work of a data indexer and request proof of work.
Although The Graph has been hugely successful, numerous competitors have emerged, offering faster and more efficient indexing services for the evolving blockchain world.
When The Graph first launched, the world of Web3 was a very different place, with the vast majority of dApps only needing to access data from EVM chains and ERC-20 transactions. However, these days it’s very different, and many dApps don’t limit themselves to a single blockchain, and some require more than just blockchain data.
This led to the rise of more versatile indexing platforms such as Goldsky and Alchemy, which rapidly gained prominence in the market. However, there are concerns regarding the centralization of these newcomers. When Alchemy acquired a subgraph hosting service called Satsuma last year, it tripled its performance but it did so at the expense of decentralization. Moreover, one of the most popular decentralized alternatives, called NXYZ was ultimately acquired by the centralized data warehouse company Snowflake Inc.
For developers who believe in the ideal of a Web3 built on decentralized foundations, centralized data indexers simply won’t do. Fortunately, solutions such as SQD, formerly Subsquid, have emerged to change the narrative again.
SQD is a fully decentralized blockchain indexer that’s known for its incredible speed, flexibility and cost-efficiency. It supports multiple blockchains and has shown itself to be around 100 to 1,000 times faster at blockchain data retrieval than The Graph, enabling requests that once took weeks to synchronize to be resolved in a matter of hours or even minutes. Moreover, because SQD is fully decentralized, it can help reduce dApp infrastructure costs by around 90% compared to traditional data service providers.
Another distinction of SQD is its modular architecture, which facilitates simplified integration with various tools, including Web2 data sources. This capability makes it one of the most versatile and useful blockchain indexing tools for a wide range of dApps, including those that require real-world information.
The Keys To The Kingdom
Blockchain indexers are quickly emerging as an essential infrastructure tool for dApp developers, addressing the limitations of traditional methods for querying blockchains and retrieving data from them.
By harnessing the capabilities of powerful blockchain indexers, developers can eliminate latency within their dApps and reduce their overall operating costs, while making them more functional. Platforms like SQD act as the gatekeepers to the blockchain, providing dApps with the keys as well as the tools they need to access, transform, organize and query the information they hold within.