1

I am running a local geth node (--fast --rpc) and executing the following code in a local Nodejs program:

https://gist.github.com/admazzola/79daee3b41e47596e36286464b1d98dd

Strangely, it works but that loop runs extremely slow and each function call takes about one whole second to resolve. Once it gets up in the 2 or 3 thousands it usually locks up or freezes up and I don't know why that is. Is there a better way to pull all of this data and store it in a dictionary or is this the best way? Why is it so slow if I am just gathering local data?

2

Is there a better way to pull all of this data and store it in a dictionary or is this the best way?

It would almost certainly be faster to parse the data directly from the .ldb files in the chaindata directory, and then cache it. To do that would require you to parse it, which itself requires knowledge of how the data is stored.

If you're using Node.js, you could take a look at "Exploring Ethereum's state trie with Node.js". (Node.js is probably not the best language to use for this.) Note that the code in the tutorial is slightly out of date. I've included a tweaked working version below.

var levelup = require('levelup');
var rlp = require('rlp');
var trie = require('merkle-patricia-tree');
var db = levelup('chaindata');

// The genesis state root.
var stateRoot = 
  'd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544';

// ERROR: AssertionError: Invalid root length. Roots are 32 bytes
var stateTrie = new trie(db, stateRoot);

// Note: We're doing everything using binary encoding.
db.get(new Buffer(stateRoot, 'hex'), {
  encoding: 'binary'
}, function (err, value) {
  console.log("Printing the entry for the State Root")
  var decoded = rlp.decode(value);
  console.log(decoded);
});

// Gav's address.
var gav = new Buffer('8a40bfaa73256b60764c1bf40675a99083efb075', 'hex');

trie.get(gav, function (err, val) {
  console.log("Printing the contents of Gav's address")
  var decoded = rlp.decode(val);
  console.log(decoded);
});
  • the stateRoot should have 0x prefix to avoid AssertionError: Invalid root length. Roots are 32 bytes – medvedev1088 Feb 20 '18 at 10:44
  • Also the address needs to be hashed before looking it up in the trie. A working example is here ethereum.stackexchange.com/questions/40254/… :) – medvedev1088 Feb 20 '18 at 13:23
  • 1
    Thanks! Let me poke around and remind myself how this works. There's almost certainly a problem with the above code using the old version of LevelUp for a start... (as in, it wouldn't work now with the newer versions of LevelUp and LevelDown anyway). I'll defer to your code from now on :-) – Richard Horrocks Feb 20 '18 at 18:58
1

I found a way to get the addresses much faster without using call! I don't really understand the merkle tree data retrieval but I will look into that too. Here is the really fast data-grabbing code (asychronous):

 var PunkContract = web3.eth.contract(contract_abi);
     var contractInstance = PunkContract.at('0xb47e3cd837dDF8e4c57F05d70Ab865de6e193BBB');




        let number_of_punks_found = 0;
        var punk_id = 0;

          for(punk_id=0;punk_id<100;punk_id++)
          {

            let temp_punk_id = punk_id;
              var punks = contractInstance.punkIndexToAddress(punk_id, function(err, res){
                  console.log(temp_punk_id);
                  console.log(res);

                  punkOwners[temp_punk_id] = res;
                  number_of_punks_found++;
              });


          }

and here is the whole codebase for you all to use as reference:

https://github.com/admazzola/punkpipeline

That is a nodejs bot that continuously reads and caches smart contract storage data from geth. It stores the data in memory and in a JSON file. It has a simple JSONRPC interface of its own so that other local bots can access the cached data in custom ways.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.