I figured this out after extensive research. You are right, Ethereum uses ECDSA! There are a few good npm libraries that help with this right now, one is eccrypto and the other is ethereumjs-util.
There is a 'tests' folder in that repo that shows examples of how to implement the functions named 'hashPersonalMessage', 'ecsign' and 'ecrecover'. Using these functions you can pick a public address X and send a challenge to its owner who knows the private key. The owner can sign your challenge message with their key, send it back to you, and then you can recover the public key using the signature. At that point you can fetch the public address of the public key and compare it to X to make sure its the same. If it is and if your challenge was truly secret/random, then you can be sure that the person who signed must have signed it with the private key.
There are public key encryption schemes other than RSA. One is ECDSA. See an example with bitcoin https://github.com/bitcoinjs/bitcoinjs-message
ETH is also using the same but I couldn't find a straightforward example.