How do websites like CoinGecko.com and Dextools.io get a token pair's FULL price history?

I understand that I can query Uniswap's theGraph, but that limits only 1000 entries. How do these websites display over 1000-entry trading history?

Can anyone suggest how I can go about collecting such large amounts of data? Should I use Etherscan for this?

All the trading information is shown on Etherscan already, but whats the most efficient approach to this type of data aggregation?

I'm really new to this and I'm trying to learn by building projects.

Thanks in advance!


Short answer: They are likely either running an archive node themselves (a full Ethereum node which does not prune old state), running a local indexer or both. You can do this, or use a paid service.

Longer answer: Whenever you query data, you're communicating with a server that has that data, requesting it, and then they send it over. This takes overhead and bandwidth. Many services across the internet will give you data for free, but to a certain limit. This can also true for services which cache blockchain data. The blockchain itself is permisionless, but if I independently index chain data, I can set limits on how much I want to hand out. We'll explore three ways to get that data on your own.

Archive Node: The blockchain is permissionless, meaning anyone can run a node. This means that if you run an archive node, you'll have all the data you could ever want locally. A regular full node prunes old state (though it does execute every piece of state from genesis), but an archive node keeps everything. These can be quite large - I think >4TB is the current state (late July 2021), dyor.

Local Indexer: If you don't have the hardware/capacity for handling an archive node, you may want to consider running an indexer. An indexer can query nodes on the network for the data you want and index it locally for you. You can write your own program to do this, or use a suite like TrueBlocks.

Paid Service:

Should I use Etherscan for this?

Etherscan's free tier limits to 5 requests per second (source), so it will be slow. (1000 requests would take over three minutes, for example.) The page linked above says that you can contact them to upgrade your plan, so it is possible that they can arrange a paid service that gives you the access you need. There are also API services like Alchemy which specialize in providing APIs for blockchain data. Services like Alchemy generally have a free tier, but if you are looking to scrape a significant amount of data, you will likely be paying for it. Please investigate pricing before using.

Hope that helps!

  • 1
    Wow thank you! I didn't think I would get such a well written reply... I will look more into local indexing. But what other resources do you recommend on learning more about Ethereum, or just about the block chain? As a beginner to dApp's, I would not have known about open source software like TrueBlocks. Where, or how can I learn more about decentralized technologies like TrueBlocks?
    – tomi lui
    Jul 26 at 1:40
  • hmm, I can't believe I'm doing this, but I don't have a better general, all-purpose solution than Twitter and Reddit. If you start following projects/builders you believe in, it will help you find more like them
    – Linum Labs
    Jul 26 at 8:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.