There are a lot of gnome themes out there that mimic the OSX GUI with varying levels of success. Basically, you will need a graphics card with linux 3D support and will install compiz, emerald theme manager and the cairo dock application. Here's an example of one such attempt:
http://maketecheasier.com/turn-your-...ard/2009/01/08
If you google "ubuntu OSX theme" you will see many more.
Your main problem will be with your graphics card as linux support for ATI graphics card is well behind Nvidia. For that reason, I haven't used ATI graphics cards for some time and am therefore not that familiar with current level of linux ATI support. I would recommend installing ubuntu again and see if you can get decent 3D support for your card. Ubuntu won't install the proprietary ATI drivers by default but you can do so after the initial installation by going to System>Administration>Hardware Drivers which will tell you if there is a proprietary ATI driver available for your card and offer to download and install it for you if there is one. You will need that proprietary driver to get any kind of decent 3D support which you will need for the GUI effects necessary to mimic OSX.