When it comes to Ethereum, there are two things you got to consider unlike most cryptocurrencies: Your download speed, and your processing speed. That difference you're seeing is most likely because your node is stuck processing "busier" blocks that carry more transactions and contracts. Moreover, Ethereum has had to deal with spam attacks before, and your client still needs to process the transactions related to these attacks at some point or another.
As a reference, I was unable to get much luck with geth at all, as my connection is not too reliable and as a security measure, upon the tiniest sign of trouble, the fast sync will be cancelled and geth will fall back to full block verification, to the point I ended up using Parity in Geth compatibility mode so I could keep using Ethereum Wallet.
parity --geth --warp took just about two days or so to get the headers for the latest blocks so I could at least see the latest transactions, and around a week to process all blocks on my measly 3Mbps connection.
