From what i know latest driver won't work with claymore (and other miners). You need to revert to the 18.04 ones. Here how it worked for me on Ubuntu 20.04 with RX580 8GB (same setup as you)
make sure you have hwe installed
sudo apt-get install --install-recommends linux-generic-hwe-20.04
make a backup of /etc/os-release and edit it.
sudo cp /etc/os-release ...