The reason my Vyper contract caused a Ganache error, is because the vyper compiler puts a \n character at the end of the bytecode line in the bin output file. Web3j used that line as is.
When I remove the \n from bytecode output and create a contract wrapper class, Web3j successfully deploys it :-)