If all miners removed the difficulty bomb and basically reject the fork (or introduce a new one by removing it) then there would still just be one chain as there is nobody else creating any other chain. In this hypothetical scenario, if the miners were the only ones and all other node operators would still have the bomb in place, then these nodes would simply reject all blocks and they would not see any new blocks ever again. In such a scenario you would create massive financial incentive for someone to switch on their miner, mining a version of the chain with the bomb in place and thereby getting all block rewards of the chain that is widely respected by everyone except for the other rebellious but lonely miners.
In a more realistic scenario in which only a few miners chose to remove the bomb while the others keep it in, then we would have a classical fork scenario with two chains. Now the minority of miners that chose to remove the bomb will quickly find out that the coins which they mine will not be exchangeable on their favorite exchanges because their accounts simply did not receive any ether on the other/main chain on which they did not mine.
Therefore, unless there is at least some sort ecosystem-wide critical mass including miners, node operators and (importantly!) exchanges (as seen with Ethereum Classic post DAO-hardfork), the system should incentivize everyone to play along the main chain and quickly abandon any minor forks.