Ever since the release of our open source single binary “dfuse for EOSIO“, we have received many requests to explain the foundational architecture that it encompasses. In this video, Alex explores the deepmind instrumentation used to exfiltrate data from nodeos, as well as the dfuse data model and the protobuf package it uses. 



Hello everyone. In this video of the dfuse architecture overview, we’re gonna dive deeper into the deepmind instrumentation and the dfuse data model. 

Now, quick introduction to deepmind. deepmind is the modification — that secret sauce, remember — the modification that we made to nodeos to be able to take data out of the nodeos node and into the dfuse platform, the dfuse data model and services. Which all flow through gRPC endpoints, and all use protobuf definitions, we’re gonna dive into that a little bit. 

Now, if you run nodeos, with –deepmind enabled, you will see a lot of information being spewed out and let me show you a little bit how this looks. Because it might be intimidating, but I’ll give you the purpose for that. So if you look at the terminal here, we’re seeing a sample output. And these look even more intimidating when they scroll like that, crazy in your face. But basically, they’re line-based and they exfiltrate all of the things that happen during execution. All the transaction traces, the database changes… And they’re all then, through the reading process, which we call mindreader, linked through that standard output, they’re read. And we slowly build, well, as fast as possible, build a protobuf object, or a Go object, where we stitch these things back together.

Let me show you the code there, it’s called the ConsoleReader. This is the ConsoleReader that slowly builds a block and reads these lines and then accumulates all the transaction traces and puts them in the order of the block. And then, this block object is what flows into the whole system through gRPC, through protobuf definitions. Every system is a streaming gRPC endpoint, so the merger, the relayer, mindreader is the first server of blocks that flows out to eosws, FluxDB, the search indexer and all these things, everyone is fed with blocks in this format, this profobuf. Which is very neat, it has a lot of extensive tooling around. You can use it to plug into all sorts of different languages.

This is the data model and also let me show you quickly the proto definitions here. This is in the package proto-eosio, all the things that are specific to EOSIO and the dfuse stack. There are things that are common to the different implementations, like the search engine calls, and bstream machinery, which we’ll get to in another video. The proto-eosio are things that are specific to EOSIO. You know, the block, and for those who are already well versed in EOSIO you’ll see those producer signatures, this is known stuff like the block state. 

And there’s some additional things, like the transaction, the implicit transactions, which you don’t normally see. The rate limiting variables that are extracted in addition to the rest. So these are not things that nodes normally spew out. You don’t have APIs to read these things, but with the instrumentation we built, we can get that, and take decisions that we normally wouldn’t have the information for, like rate limiting. 

And the most important thing is the transaction trace. These things are the actual meat of the execution of the chain. And in there, the important bit, action traces, where we actually see, you know, things that you might have actually seen before. The receiver, the action, the time it was elapsed, and whatever side effects it could have caused. The transaction trace will also hold, sort of in a tree fashion, all the actions that occurred, who triggered what, and all that. This is what you get when you query the GraphQL endpoint when you go to eosq and you see those nested things. This is the transaction trace, there you have a list of such transaction traces and a block, and the block is the top-level component that flows through the system. 

So, hopefully this gives you a little better understanding of what deepmind is. And also, let me show you this PR, here, that’s the PR that merged inside of nodeos. So it’s coming to the version 2.something, probably higher than 2.0. It’s not in the release by Block.one in 2.0, we do have the releases in our repository so you can get the debian, or for Mac or Linux. The release that are already instrumented with the deepmind. It’s merged into the nodeos codebase now, so you will get it with the next revision. I’m not sure the version that is going to be out by Block.one, we’ll see that. But, it’s gonna have deepmind directly inside it, so you won’t need to install it separately. And you’ll be able to work with dfuse on top, get all the benefits there. 

Ok, hopefully, this is helpful. If you have any comments put them in there, in our Telegram channel, and we’ll see you next time.