Three ways to dramatically speed up your Joomla! site:
1. Enable Joomla!'s caching. There are also caching features that come with many of the more modern templates such as rockettheme's gantry cache,
2. If you have Firefox, install YSlow (
developer.yahoo.com/yslow/) and go through the list of results it produces. A lot of the improvements are to do with server and client caching improvements which can result in a dramatic increase in performance (and takes the pressure off of your web server),
3. Consider running nginx (Engine-X) for serving up your static content (images, js, css). it is faster than Apache and should result in better load times.
Ultimately one of the biggest performance hits is taken when the web browser has to render HTML. Be sure to use templates and extensions that implement clean HTML (I.e. all styling is in external stylesheets, html uses divs instead of tables for page layout, etc) to get the best possible load times.
There is also now a move to integrate various 3rd party APIs and widgets into web sites such as google analytics, sharethis and twitter status updates. These usually require the web browser retrieve js and css files from another server that hosts the widget code which will also result in longer loading times. Try, if possible, to limit the number of these widgets on your web site.