The block number will always be correct by definition: It's number x in the chain, because it's chained on top of x-1.
However, as you say
block.timestamp can be gamed a little bit - or with the cooperation of the economic majority of validating nodes a lot - which also means that the relationship of
block.number to actual time can be gamed. So if you don't trust that a
block.timestamp of 2017-01-01 will really be approximately 2017-01-01, you can't rely on counting the blocks that are supposed to be mined between now and 2017-01-01 either.
PS. People may be able to give you more helpful advice if you tell us what purpose you want to be secure for, and what you want to be secure against.