What was the reason to invent the Ethereum Virtual Machine (EVM)? Why not use an existing virtual machine like the Java-VM. The big advantage would be that the existing toolsets for Java development could be used.
So the important point in your question is why a new VM and not a java VM. ok let's choose a Java VM instead, what do we get? :
1- complex and voluminous Bytecode => how to store it, in the blockchain for which cost? (Knowing that a single java method can have a size up to 64KB) thus such VM or language isn't space-saving.
2- useless features and security concerns: Network access,I/O stream, File W/R ... =>each of these features implies big security issues. think of it, you can write a code which ping (of death) another machine or access protected files, or even steal the miner's private keys. Even sandboxed a flaw could occur and file "write/read" feature could break the whole system's security. so we'll need to get rid of all these features, which is a hard task to achieve for a licensed VM. We need to remember that a Blockchain VM should be isolated without the capacity to communicate with the external environment.
3- if you have a JVM, you would use Java or another JVM language, which are not adapted to the blockchain context (lack of determinism, memory and hardware access, higher execution latency,floating-point, etc...). Imagine you have a Java bytecode with a rand(), what would be the result and how to reach the consensus then?.
4- Weak DDos resistance (how would you set a gas-like system in a complex VM like Java VM? ).
3- JAVA VM is a licensed Sun product, so you can't customize it to integrate it to the Ethereum's environment (for example how would you calculate the gas cost to avoid Dos attacks?)? to overcome this problem you need to write your own Java VM which is a complex task read: https://softwareengineering.stackexchange.com/questions/71561/java-and-jvm-license.
Besides, you can check out this answer about why to create a new contract language : What is the merit of creating new smart contract languages like Solidity instead of using other languages?
Vitalik Buterin gave a presentation about Ethereum to Hyperledger April 28 2016 and had this slide:
● Small code size (so that very many contracts from many users can be stored by one node)
● VM security designed around running untrusted code from arbitrary parties
● Multiple implementations (for cross-checking, and to mitigate developer centralization in the public chain)
● Perfect determinism (for consensus)
● Infinite loop resistance
- This itself must be accomplished perfectly deterministically; timeouts are a no-go
Vitalik clarified the third point as the developer centralization observed in Bitcoin, where a single implementation is deemed the protocol.
There was no time in the presentation available to discuss the evaluation against existing virtual machines.
Smart contracts need to have an absolutely deterministic outcome, because multiple machines will be running it. It also needed to store state in the blockchain. Because no VM met those requirements, a new one had to be created. EVM was created to meet all the requirements while allowing as much flexibility as possible.