I am wondering why it was necessary to develop a whole new programming language for smart contracts. Stratis for example claims that C# can be used to write smart contracts. Couldn't Ethereum implement a compiler for any programming language? What is the advantage of Solidity over say C# etc?
Couldn't Ethereum implement a compiler for any programming language?
Yes. But it'd have to produce EVM (Ethereum Virtual Machine) bytecode.
With regards to the rest of your question, this should cover it: How was Solidity conceived and what design tradeoffs were made when building it?
Why invent the EVM? That's covered here: What was the reason to invent the EVM?