I understand that if I break/lose my Ledger Nano S, it's possible to recover it if I get a new Ledger, and using my seed combination. But then it means that any hacker can do the same, no?
Yes, your seed regenerates the private key on your device, and it can in fact do so with any Ethereum wallet that supports seed regeneration of a private key. Anyone can do the same, assuming they have access to your seed, so it is up to you to keep the seed in a safe place.
If you are concerned with security, there are 2048 words to choose from in the seed, and your seed has 24 words. 2048^24 = 29642774844752946028434172162224104410437116074403984394101141506025761187823616 possibilities. That's even greater than the key space of your private key.