As far as I know, that type of VJing is done by rendering 3D graphics in programs like 3D Studio Max or Maxon Cinema 4D. Your best bet would probably be to start there and play around with them for a bit.
3DS Max info:
http://usa.autodesk.com/adsk/servlet...&siteID=123112
Cinema 4D info:
http://www.maxon.net/pages/products/...ewinr10_e.html
As far as free alternatives go, you're pretty much on your own there. I don't know of any offhand.. ^_^;