JPGs
The JP(E)G format, which stands for Joing Photographers' Expert Group, is, as the name suggests, quite good for compressing photographic images for the web. JPGs use a lossy compression scheme called Discrete Cosine Transformation to average out a block of pixels. "Lossy" means that in compressing the image, information is lost, so the JPG is never as good as the original. For this reason, you should always keep an uncompressed version of an image, and only make a JPG copy before you add it to a web site.
One feature that makes JPGs unique is that you can determine just how much compression you want to use: the more compression, the smaller the image (in file size), and so the faster it is to download. At the same time, more compression means worse quality, so high quality means sacrificing some speed.
Let's look at some examples to see how this works in practice. Here's as pristine a quality JPG as we can get, 100% quality:

At 100% quality, the image is 204kb, which will take about 37 seconds to download on a 56k modem.
At the opposite end of the spectrum, the same image at 0% quality compresses down to 13kb, and will take about 3 seconds to download:

Clearly it looks worse, by how, exactly?
What's happening here is that the image is divided into squares eight pixels high and down, and, depending on the degree of compression set, average out the colour values of those pixels. Let's look at a close-up to see this. First, the image is divided into a grid:

Then, each of the squares in the grid moves towards an average, depending on the amount of compression. Let's zoom in on the 100% image by a factor of 800%:


Compare it to the 0% quality image:

If we place the overlay lines showing the grid of 8 by 8 pixels—

—you can see how batches of pixels lose detail as the colours move closer and closer together.
But if these are the extremes, there's a wide middle ground to work with. A 50% quality JPG might be a good compromise:

At 52kb, it fairly small (about 10 seconds to download), but still looks pretty good—here they are in sequence, from 0% to 50% to 100%:



JPGs aren't perfect, though—the very scheme that makes them work so well on photographs, where the general idea can be more important than the specifics, makes them somewhat unsuitable for large amounts of type.
Here's a different kind of image: a logo in GIF format:

This image is 4.2 kb in size. Not too bad, especially as the uncompressed Photoshop file it comes from is 160 kb.
Now let's look at the same image as a series of JPGs, first at 100% quality

It looks okay, but the image is a whopping 23 kb: almost five times as bit as the GIF!
Let's knock it right down to 0%:

It's now down to 2.6kb, but boy it looks awful. Because the JPG scheme is all about averaging, it works well with photographs or things that are continuous tone, where you don't notice subtle differences in tone so much. But here, where there are essentially two colours, blue and white, the JPG introduces all kinds of intermediate colours, and really destroys the type.

Fortunately, for logos, type, or flat-toned illustrations, GIFs are available, as they compress quite differently, and respect shape and colour in a way JPGs don't.
One thing to keep in mind, though, is that you need to balance not just a single image's quality, but how a whole site is put together. How many images are on a single page, and what's their cumulative file size? Is there an image that shows up on every page (could be, as part of a global interface like a navigational button)? If so, you might want to make sure that image looks good, while an image that only shows up once can probably be whittled down a bit.