Backups are a Good Thing, but making a usable backup of a database driven content (such as a Joomla site) is a bit tricky. You need a consistent copy of to both the source files and database tables - fortunately there are extensions that can help.
To make backups of my Joomla site (and other sites I'm editing) I use the Akeeba backup system [1]; there are other site backup extensions that could do a similar job.
Installation
Installation follows the usual Joomla extension install process. Once installed there is a component configuration wizard that sets up some server-specific values (such as where to store the backup files).
The paid version has some additional features which are useful for highly managed environments where things like automated backups are important.
Note that although this extension currently (Oct 2012) supports PHP5.2 and above it is moving to only 5.3 and above - PHP 5.2 was end-of-life in Jan 2011 with the last support release in Dec 2011 so this seems reasonable to me.
There is an integrated update mechanism so it's easy to keep up with the latest release.
Running the backups
The extension adds an entry to the Components menu: 
If you use the default profile, making a backup takes only a few clicks (and a little patience while the process runs):
Click the usefully named 'Backup Now' button in the Akeeba control panel 
Enter a description if you like then start the process by clicking the 'Backup Now!' 
Wait and watch the status messages (if you're easily amused) 
After a few minutes depending on the size of your site and speed of your server your backup is done 
By default, the backup process creates a single 'jpa' archive file, although you can create a zip file if you prefer (stick with jpa unless you have really good reasons not to).
You can use the built-in file links or any FTP program to transfer this file to another site for safe-keeping or restoration.
Testing the backups
Making a backup is only part of the process. Your backups are useless if you can't use them to restore your data.
The Akeeba backup system has fully documented restoration process that lets you take your backup archive (jpa file) and restore either overwriting a site or creating a completely new one. If you don't know that you can restore your site your backups are only giving a false sense of security.
Other uses - Site Development and Staging
Because you can restore an Akeeba backup to a different server, you can use the this process to copy an entire Joomla site from a staging server to a live one, which lets you experiment with new features and designs without risking accidentally changing your live site.
References
Akeeba backup home page
