Currently building a smart contract that will basically receive money from users and users can withdraw what they've put into. I want the money held by the smart contract to be only USDC (+ USDT maybe, but let's first focus on one token).
How can I do that please ?
PS : as you guessed, I don't own the token (similar questions have been asked but they all assumed you have created the ERC20 token).
There isn't a way that you can stop someone from sending arbitrary coins to your contract's address. "Sending" coins is essentially the same as writing someone's address in a database next to a number. You can't stop someone from simply writing values on chain if you don't own the logic behind how that data is written.
If you take a look at the ERC20 token that OpenZeppelin provides, you will find that the only addresses that are blacklisted are the zero address, on lines 231 & 232. Otherwise, there is nothing that stops the user from transferring from one arbitrary address to another.
As Majd TL suggested in the comments, you can have a deposit function that serves as the "correct" method of depositing coin via approval + deposit.
