Subscribe to changes in contract balance (Web3j)

Question

Using web3j I would like to subscribe to changes in a contract balance i.e deposits and withdraws.

I would like to base this subscription over transactions (and not events) since the contract does not emit deposit/withdraw events.

The deposit part is pretty easy since I can use the tx.getValue() filter:

Observable<Transaction> res = web3.catchUpToLatestAndSubscribeToNewTransactionsObservable(DefaultBlockParameterName.LATEST);
res.subscribe(tx -> {
            if (contractAddress.equals(tx.getTo()) && tx.getValue().compareTo(BigInteger.ZERO) > 0) {
                handleDepositeTX(tx);
            }

The withdraw (contract sending ether to some other address) is not that straight forward. I guess I can maintain the contract balance before and after each transaction but this doesn't feel right/safe:

    res.subscribe(tx -> {
        if (contractAddress.equals(tx.getTo()) && tx.getValue().compareTo(BigInteger.ZERO) == 0) {
            EthGetBalance currentBalance = web3
                    .ethGetBalance(contractAddress, DefaultBlockParameterName.LATEST)
                    .sendAsync()
                    .get();
            BigInteger diff = prevBalance.subtract(currentBalance);
            if (diff.compareTo(BigInteger.ZERO) > 0) { // withdraw
                handleWithdrawTX(tx);
                prevBalance = currentBalance;
            }

Is there a 'right' way of subscribing to a contract balance change? How can i know if as a part of some transaction ether was transferred form some specific contract to some other destination?