The prefix has no intrinsic connection to the signature or validation function.
ecrecover is simply recovering the public key (and from there the address) used to sign the 32 bytes of data you're feeding it.
There is a convention in Ethereum for signed messages (ie not signed transactions) to prefix some data to a message before you sign it. This is done to make it harder to trick people into signing data that they don't intend to; For example, if you maliciously prompted someone to sign a transaction transferring the funds in their account to yours, and tried to get them to sign it thinking it was some message intended for some other system, your attack would fail because the client that signed it would prefix this arbitrary data to it, and that would stop it being a valid Ethereum transaction.
By convention the data used for this purpose in the Ethereum world is
\x19Ethereum Signed Message:\n32, but anything else would have worked equally well. The same data would work for the same purpose with any other signature scheme, elliptic-curve-based or otherwise.