Ever since the release of our open source single binary “dfuse for EOSIO“, we have received many requests to explain the foundational architecture that it encompasses. Alex walks you through the 17 micro-services that make up “dfuse for EOSIO” and what they each do. There will be a video to explore each one in depth, so be sure to subscribe and hit the notification bell to keep up to date!
Today I am really excited to start a series of videos on the dfuse architecture. In this series of videos, we’re going to go through an overview of all the components and things that make up a dfuse instance, a dfuse setup.
You guys have access to dfuseeos. This contains all the components that will run on your laptop and that can be deployed also on some clusters, where each of those components can then be deployed separately, scaled independently. And I’m going to go through all of those components so that you get a better understanding of the general idea of dfuse architecture.
In this first video I’m going to just give you the list of components. We’ll have a video specific for each of them. So this is just a basic overview. Let’s start right away.
So when we start a new project in dfuse here, we can say `dfuseeos init`. And then if you answer yes to this question, it’s going to set up the most complete configuration for the setup for a local instance. And it contains the configuration that basically says when you start, boot all of these components together in the same process. And they’ll all be using the default values from the config flags, all be interconnected. But they’re using TCP sockets in between so you can then easily split them together, just change the flag so that they now communicate together. And then you can start scaling your deployment. So these are the components.
You can have `dfuseeos start`, you’ll have a list of all the available components. You can start one of them, two of them, all of them. And then you have all of these flags that are specific to each of those applications. The apiproxy, and then there’s a bunch of flags in there for you to configure. So I’m going to just go through the list, and tell you approximately what it does. There is going to be a video on each of them.
So the first element here — I’m going to put them in batch. The search system is composed of four components. So search-live, search-forkresolver, search-indexer, and search-archive. The indexer is the thing that does the work to take incoming data from the chain — we’re going to see what produces that data in another video — takes the data, indexes it, puts an index so that the search-archive can consume it. And then the search-live is the thing that indexes blocks real-time and then applies those queries in real time. And then search-forkresolver is the thing that allows you to renavigate forks after the fact. Giving the high levels of guarantees that dfuse provides. Now that’s it for search.
Then there’s the mindreader, and sort of paired with node-manager. These two components are things that wrap nodeos to execute it and pilot it. I’m going to have a video on mindreader especially. So one is going to run on the producer node, and one is going to run on the reading node. In a setup where you are syncing another network, you won’t run the producer node, but the features are there. We’ll have a video for that.
And then we have the merger. Merger is the thing that is designed for highly available setups. It takes the blocks that are produced by many mindreaders, fuses them together and stores that on the side so that any process can feed blocks, similar in a sense to what state-history does.
But this thing combines all forks that are seen, so you’re always able to navigate out. We’ll get details about that a little bit later. trxdb-loader is the process that takes in any information extracted from nodeos, and writes it into sort of a simple key value store. it’s basically the store for blocks, transactions, transaction traces, all of these things. trxdb-loader feeds it in.
Then we have eosq. eosq is the block explorer that you’ve come to love, the deepest, gives you a lot of information. Basically a looking glass on the blockchain. It comes built in to dfuseeos, if you boot a local thing you’ll have it. All the streaming capabilities, all the depth of the data for your debugging purposes, that’s pretty cool.
Now, apiproxy is a simple thing that we put in front just so that you can route to the different processes, that looks like a real dfuse deployment. When we give you an endpoint for dfuse, it’s like mainnet.eos.dfuse.io. Well with the apiproxy, you have this similar thing, and it routes to the different components without you needing to run an nginx, or whatever thing in front. Even though you’re separating the services. So, a small built-in nginx.
fluxdb is an awesome piece of engineering I think. Very special, it’s very intertwined with blockchain. It’s the database, it gives you the state of the blockchain at any blockheight in history. Specially crafted to give you that with a special indexing strategy to give you it at any blockheight.
blockmeta is sort of a spinal chord of the whole infrastructure. blockmeta is always aware of the state of the chain, and can help the different systems to bootstrap to know “am I in catchup mode?” or whatever. So blockmeta is a simple service that we hit. Is that block reversible? What’s the highest block? Small thing, spinal chord, everything syncs up together to know the state of the network.
Then, dgraphql is the GraphQL interface. Calls the other systems. Basically does the interface, exactly that. The relayer here is a component to aid is sort of scaling your infrastructure. We’ll see the mindreaders that provide block data from the first layer. Relayers are sort of a fanned out mechanism for propagation of blocks — just topologic. We’ll have videos about that.
And then abicodec is a service that is dedicated to encoding and decoding JSON blobs, in terms of the ABI. It knows the ABIs at any blockheight, it’s able to decode and encode. Small dedicated service so that when you need to do these things, we can scale it separately. abicodec, coder, decoder.
And eosws is one of the first projects we started when we started dfuse. It has all the websocket machinery, it does some forwarding to other services, it contains all the transaction push guarantee features. There’s a bunch of other things. We’ll have a video about that one too.
The dashboard here is something specific to dfuseeos, the binary. It’s not something we ship in production. It’s just to understand the apps that are running when you’re running on your laptop. Are they running? What’s the state? Whatever. A debugging tool for the single binary.
I hope this is helpful just to debunk, and demystify a little bit the components. And we’re going to go very deep into each of those in this series of videos that we’re going to go through.