I'm not sure what you mean by using the contract ABI within a contract. I think you mean when we import a contract into another contract and access an instance of it by creating a reference to it eg contractInstance = externalContract(contractAddress).
Ultimately the reason there are return values when we have imported a contract is because we know at compilation what type and size of data each function call is going to return. We can then put the return data in the correct variable type. However, when we call a function on an address directly, we don't know what the type and size of return values are because these details are not stored on the blockchain. The function just returns raw data. This means we don't know what type of variable to store the return value in after calling that function, so it just isn't allowed.
As a side note, since the Byzantium fork you can actually get return values from doing address.call() if you execute some assembly instructions directly after it. Specifically the returndatacopy() instruction which can be seen in the doc. And an example usage can be seen in an upgradable proxy contract I wrote (Note the example uses delegatecall but it should behave the same as using call). This approach still only returns raw data, it doesn't create a type of variable for it to live in.
