You'll need to the encrypt the data before storing on IPFS. But its not efficient to use ECIES encryption (which uses Ethrereum keys) on large data as its slow and Metamask will often hang. A good strategy would be to encrypt the data using a symmetric encryption like AES256. You can use something like CryptoJs for the same. Then you can encrypt the AES256 key using the Public Key of the recipient's Ethrereum address.
To get Public Key from Metamask, you can request it via
eth_getEncryptionPublicKey (docs). Or if the recipient has at least one transaction on Ethrereum, you can recover public key from the transaction data like this.
Now that you have the recipient's public key, you encrypt the AES256 key with it using ECIES encryption. Now you can add both the encrypted data (encrypted using AES256) and the encrypted AES256 key (encrypted using ECIES) to IPFS. You can do this using various ways, like creating a text file of the encrypted key and zipping it along with the encrypted data so you get a single file. Or add it as separate files, or create a directory.
To get the data, the recipient will have to download both the data and encrypted AES256 key from IPFS. Then decrypt the encrypted AES256 key using their public key. You can do this using Metamask using
eth_decrypt(docs). Now with the AES256 key, you can decrypt the data.
You can checkout this project which implemented this idea. Hope this helps :)