The aim of a hardware wallet is to never expose the private keys for the addresses it generates. The wallet doesn't want anybody to ever see those private keys. Instead of trying to determine if you're you, or if you're a hacker, this system just assumes that you're a hacker.
Why assume that can be trusted? If you're you, you shouldn't need to know the private keys to access the assets held on the hardware wallet's addresses when you've got the device's pin number?
With this system, the only time you're ever privy to information that can grant you access to the assets without the device is when you first set it up, and are given the 24 word recovery phrase.
NOTE: Definitely write that phrase down, and keep it very safe and very hidden. If you lose that phrase, and then accidentally drop your hardware wallet into an ocean, you're never accessing the assets on it again.
With this system, you can't access an address on a hardware wallet that was not generated from within that wallet, since that would oppose the intentions behind the device's design (keeping even you from knowing the private key). If you want to access assets on an address that you've used prior getting the hardware wallet (paper wallet, Metamask, dsektop wallet, etc.) then you'll have to send those assets from one address to another.