[freakalad]

I think it might be best the desktop is optimized for an ultra-light WDM, without all the extra crap you won't need, like office, mail, etc.
If users want a fuller desktop afterwards, then I would suggest they install Unity and/or Gnome Desktop/Shell themselves - it's literally 1 command from the CLI.

With my present installation (not a XBMCnubtu, but a skinny with X only) I've managed to set it up in such a way that additional VTTY's run other application, such as GMPC or Boxee - just for the hell of it (it's actually very cool).
I might do something similar with Ubuntu TV and/or a mobile interface.