Are you sure that's more than 20 score? In low ranks like 300+, you can find that a different of just 10 score can drop a player by dozens or even hundreds of places.

(Remember, 1 score does not equal 1 rank place)

If it is, consider that you could have lost score from more than just that crystal. If you recruited units and dropped a multiple of 100 from your population, that's another 10 score gone. If you had any conquers then you've also lost 10 score for each conquer you lost.