There are some things you can do to speed up XBMC on the Raspberry Pi, but you have to accept the fundamental fact that although GUI performance is now pretty good in recent versions of XBMC for the Pi in terms of scrolling, screen refresh and so on, (and that video playback is excellent - I just did a test today where the Pi could play 70Mbps 1080p H.264 without any frame drops!) that anything that takes raw processing power will always be slow on a Pi.
In particular, searching for listings in certain add-ons can be slow. Typically you will have a 2-3 second delay between selecting an item in a list and the next list coming up, and anything that is building a list of movies etc from data online such as a search could take 5, 10, 20 seconds or more while a PC might do it in 1-5 seconds.
Having said that there is quite a bit you can do to speed things up. Personally I found one of the biggest improvements was to start using the new F2FS filesystem instead of ext4. As far as I know, Raspbmc is the only distribution offering this, so while I know you're using OpenElec, you might want to give Raspbmc a try.
http://www.raspbmc.com/ Raspbmc can now use F2FS on both SD card installs and USB installs. (bear in mind most other OS's cannot read F2FS formatted disks)
Overclocking is of course essential for obtaining best performance - every Pi has a different overclock limit beyond which it becomes unstable, my Pi (with Samsung memory) can run 100% stable on "Super" over clock in Raspbmc which is 950 cpu, 450 core, 253 gpu, 450 sdram, and 6 overvolt. On Raspbian I can run it on High overclock (950 cpu, 250 core, 450 sdram, 6 overvolt) but it can't even finish booting on Turbo (1000 cpu, 500 core, 600 sdram, 6 overvolt)
By the way, you'll read a lot about SD card corruption when over-clocking with people suggesting to use a USB install if you over-clock - this is no longer necessary as of the firmware released in December. The current version of Raspbmc contains the fixed firmware so over-clock related SD corruption seems to be a thing of the past. I'm not sure if the current OpenElec contains it but I would imagine so.
The next thing is to use a lightweight skin - you're already using Amber and that's probably the most light weight but still good looking skin, so stick with that.
A USB install can speed things up, but only if your USB drive is faster than your SD card - and that's not always the case! Some cheap USB drives are very slow (particularly write speed) so it's best to do a speed test of both SD card and USB drive before deciding which to use. See here for details:
http://www.raspberrypi.org/phpBB3/viewto...26&t=31925
Raspbmc has a check box in the installer which you can use to very easily do a USB install if you decide to go that way - it does all the partitioning of the USB drive for you.
If you're currently using a wireless connection it might be worth trying an Ethernet connection if at all possible for best performance.
Another thing that can speed up the GUI (ironically) is to enable 1080p GUI mode - by default XBMC on the Raspberry Pi only runs the GUI at 720p and scales it up. (Video plays at native res however) For some reason this seems to cause GUI animation to stutter and I find the GUI actually runs more fluidly if I enable 1080 mode for the GUI. In Raspbmc this is accomplished in the raspbmc settings addon by choosing "Remove UI res limit". I don't think there is any way to do this in OpenElec currently.
Hopefully that helps. Personally I'm quite happy with the performance of XBMC on my Pi - video playback itself is flawless even at 1080p output with high bitrate files, I can use the TV remote to control it, the basic GUI navigation speed is fast enough and fairly smooth (at least on Amber) the only real issue is that response times navigating add-ons is a bit slow, but I accept slightly slower navigation to content for the convenience of having an inexpensive, tiny, silent, very low power consumption box attached to the TV instead of a big noisy home theatre PC.
It has also been extremely stable for me - far more so than XBMC running on Mac OS in another room which crashes on me fairly regularly...the Pi is left on 24/7 and has only crashed on me 3 or 4 times in several months, and all of those "crashes" have been XBMC, not the underlying OS or the hardware, and I've had no file system corruption at Super over-clock on an SD card install.