How can I parse an Ethereum block-level logsBloom?

Question

I have been downloading blocks from EVM-based blockchains via the JSON-RPC API via eth_getBlockByNumber. In the block there is a logsBloom element which, if I understand this correctly, is an aggregate of the logs of every transaction in that block.

There are certain topics I want to search for, such as 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef (token transfer). If I can search the block-level logsBloom filter for my topics in my software, then I can decide whether to go back to the blockchain to get any of the transaction-level logs and/or receipts.

I have limited API calls to my blockchain data provider, so I don't want to call eth_getLogs or eth_getTransactionReceipt for every single transaction on a blockchain since 99.9% of them are irrelevant to me and I simply can't send a million requests per day. I need this data historically back to genesis, so subscribing to real-time, filtered events is insufficient.

Is this possible? Can I parse and search the 256 byte block-level logsBloom data for "any transaction in the block which emitted a specific topic/event that I am asking for"?

Answer 1

To check if a topic might be possibly included in one of the transactions in the block, you must check if 3 bits in the logs bloom are set to 1.

Let's go through the example:

topic: 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef
logsBloom: 0x0100000000000004000000000000100000000000000000000000000000200000000000000000000000000000000001000000000000000000080008000000000040000000002000000000040800000000000000000000200000000000004800000040000002000000000000000000081000000000000040000000001080020000200001000000000000000000040000004000000000000000000000000010000000008000000000000800008000000800000000008000000200000800000000000000200a000000200000000000100002004000000000000000000000000020000000000080000020000000800000000000004000000000000000080000004000

Get the keccak256 of the topic: 0xada389e1fc24a8587c776340efb91b36e675792ab631816100d55df0b5cf3cbc

You can easily check if a specific bit is set to 1 with ByteMask, e.g. 00001000 == 8. To check if a Byte (e.g. 0x33 or 51) has the bit set to 1, (51 | 8) === 51. Means when we OR all bits of the number A and number B, we should still get the A, as the same bit in A and B is 1

So from the 0xada389e1fc24a85... we have to find 3 ByteMasks and ByteIndexes to check in the bloom logs for.

We can refer to the source code, how the Bits are calculated which should be set to 1.

https://github.com/ethereum/go-ethereum/blob/793f0f9ec860f6f51e0cec943a268c10863097c7/core/types/bloom9.go#L139

And do the same thing:

function getNumber (hex: string, byteIndex: number, bytesCount: number = 1): number {
    let start = hex.startsWith('0x') ? 2 : 0;
    let i = start + byteIndex * 2;
    return parseInt(hex.substring(i, i + 2 * bytesCount), 16);
}

function getByteMasks (topic: string) {
    let hashBuf = keccak256(topic);
    let byte1Mask = 1 << (getNumber(hashBuf, 1) & 0x7);
    let byte2Mask = 1 << (getNumber(hashBuf, 3) & 0x7);
    let byte3Mask = 1 << (getNumber(hashBuf, 5) & 0x7);

    const BloomByteLength = 256;
    let byte1Pos = BloomByteLength - ((getNumber(hashBuf, 0, 2) & 0x7FF) >> 3) - 1;
    let byte2Pos = BloomByteLength - ((getNumber(hashBuf, 2, 2) & 0x7FF) >> 3) - 1;
    let byte3Pos = BloomByteLength - ((getNumber(hashBuf, 4, 2) & 0x7FF) >> 3) - 1;

    return [
        { index: byte1Pos, mask: byte1Mask },
        { index: byte2Pos, mask: byte2Mask },
        { index: byte3Pos, mask: byte3Mask },
    ];
}

And now, we can check our logsBloom if it matches the topic.

let masks = getByteMasks(topic);

let bitsAreSet = masks.every((byte, i) => {
    let b = getNumber(logsBloom, byte.index);
    let has = (b | byte.mask) === b;
    return has;
});

If bitsAreSet is false we know that none of the transactions in the block has the Transfer log. But if it is true, it doesn't guarantee that the Transfer event is present, just it could be present.

Answer 2

The logs counted by a node provider such as Alchemy or Infura don't count every log/block in their single call limit, just the volume of logs matching the filters provided. Further, you can run these requests in a for/each or do/while loop to scan the blockchain in a macro sense, and further examine each tokenTransfer log returned.

For you to return logsBloom in an attempt to decode it, you wouldn't really get anywhere, because there's not really anything to 'decode':

Each tx has up to 4 topics, 3 of which represent bits, and are set depending on the hash of the topic in hand. Overall, this 2048bit string of logsBloom can be examined to see if the bits match the topic hash you are looking to find - if they don't match, that block didn't include a topic hash matching the one you are looking for. And even if they do, you would want to look further and ensure the block does indeed contain your topics, since these strings may contain false positives.

Bottom line - don't worry about rate throttling, just look explicitly for the topic has you're interested in. If you do reach a rate limit, which is I think >100,000 returns per day for Alchemy, you can go pro :)