Yes you can do that, but it is not a secure and safe way to do it.
If you do not have much at stake if a malicious miner manipulates the block timestamp, it's most likely ok. Otherwise, it's not.
From the doc:
Note
Do not rely on block.timestamp or blockhash as a source of randomness,
unless you know what you are doing.
Both the timestamp and the block hash can be influenced by miners to
some degree. Bad actors in the mining community can for example run a
casino payout function on a chosen hash and just retry a different
hash if they did not receive any money.
The current block timestamp must be strictly larger than the timestamp
of the last block, but the only guarantee is that it will be somewhere
between the timestamps of two consecutive blocks in the canonical
chain.
Regarding your other questions:
- Network congestion should not impact the block time, as far as I know.
- Malicious miners can always manipulate the state of the blockchain, but only to some extent. If you wait for 15 blocks to be mined after the block you are checking (typical confirmation delay), you can assume that the block is real and accepted by all miners.