Both the 360 and PS3 controller does take some getting used to. After so many years of being a mouse and keyboard type person, you will just think in your head what the buttons should be or the flick of your wrist for an action.
Honestly I would say if you haven't already pick up a used copy of Rainbow Six: Vegas or Vegas 2. Both of those games will help swing you into getting used to controls on a pad very smoothly. I do agree with some of the quirks in Red Dead Redemption, I myself had to take some time to get used to it.
As for the Game marketplace and Arcade games, every game in the XBL arcade has a free demo/preview (actually 99.5% do) that you can download and play. From both inside the demo as well as the browse page you can pick a purchase option to buy it. It's actually very streamlined, however again being used to the PC mindset you have to retrain yourself.
All I can say is just continue to be patient and relax, your skill will come in time. Just pick something and stick with it!