On The Importance Of Inherent Structural Properties For Learning In Markov Decision Processes