If your shares are rejected, they are malformed. This is in contrast to expired shares, which were simply shares that were sent too late (i.e. the block those shares were meant for was already mined).
My guess is that your memory clock isn't stable enough to support your higher core clock. Try to first maximize your performance (regardless of instability) with your core clock, and then sliding your memory clock higher until you get a stable miner. You can check with your pool to see if your rejected shares are minimized, after gathering data for a few hours.
Once you achieve stability, you can then try to reduce the power limit or core voltage in very small increments to decrease power consumption and heat generation. This is entirely optional but can increase your fan lifespan.