Q: What is the evidence for the Big Bang?

Physicist: The very short answer is: all the galaxies in the universe are flying apart, so at some point in the distant past they must have been very close together.  It would have been so close and so dense that all the matter in the universe would have been extremely hot.
It so happens that, since light takes a while to get from place to place, we can actually see that early age of the universe by looking really far out. That ancient heat now takes the form of the “cosmic microwave background”, which we can see with radio telescopes (and hear with regular radios). We can then follow the rough history of the universe forward. By looking at distant galaxies we can even see how the universe has changed up until now.

The Hubble Ultra Deep Field survey. Although this is not a "true color" picture, almost all of the color variation originates from the "cosmological red shift", an effect caused by the expansion of space which increases with distance. This square is less than one eighth the size of the moon on each side, and it was selected because it's one of the emptiest regions in the sky.

The longer answer is a bit more specific.  Back in 1929 Edwin Hubble (of telescope fame) discovered that the farther things are away from us, the faster they’re receding.

Hubble's original data. Errors are often big in astronomy (and even bigger in astrology). The data has gotten a lot better since then. Image remorselessly stolen from: http://www.astro.ucla.edu/~wright/cosmo_01.htm

The first question that came to his mind was: “what?” followed shortly after by “what does this say about the universe in the distant past?”
If you look at the motion of all of the matter in the universe and “run time backward” you find that somewhen around 15 billion (give or take) years ago the distance between everything would have been zero.  Having everything pressed together early on implies that the universe would have been ludicrously hot, and that the universe we see today is composed of the cooled off, still flying apart, remnants of that early super-furnace.

Btw, the term “Big Bang”, an understated name coined by cosmologist Fred Hoyle, really doesn’t convey either the bignness nor the bangness of the beginning of the universe.  Other suggested names include the fairly popular “Horrendous Space Kablooie!”.

Excellent suggestion.

Oddly enough, the Big Bang isn’t a simple as a big explosion happening somewhere, followed by all the matter of the universe flying away like shrapnel.  Immediately after an explosion you’ll find that the material involved is moving in every direction, at many different speeds.  But you can say even more; the distribution of those speeds is roughly Gaussian, so the faster the speed, the less stuff you’ll find traveling that fast.

The distribution of matter resulting from an explosion. This is not how matter is distributed in the universe. Not at all.

So, if the Big Bang were just an explosion somewhere, then the distribution of the matter in the universe would follow suit: the more distant matter would be the fastest stuff, and there wouldn’t be much of it.

However, that’s not at all what we see.  Instead, the galaxies we see are distributed (roughly) evenly throughout the universe. Cosmologists call this “homogeneity”, and the chance of an explosion producing it by accident is zero.

So, the universe is expanding, and yet it’s not expanding from a single point. This, and a wide variety of other bizarre (complicated) effects, can be easily explained by “metric expansion”.  A good way to talk about metric expansion is to think of space as the surface of an expanding balloon.

Metric expansion. Even without moving, objects find themselves farther apart. Even more profound, there's no center to the expansion. Every point is as good as any other.

Draw a couple points on a balloon with a pen and then blow it up (pardon, “inflate it”).  I’ll wait.

You’ll notice that the points move away from each other at a speed proportional to the distance between them, and that the points aren’t really moving at all.
There’s a nice symmetry here in that no point is at the center of the expansion. In addition (and this is what makes it real science) the metric expansion description provides a testable hypothesis: you should be able to see what the universe looked like when it was extremely small and hot.

When you look out into space you see the heat of the early universe. Shown here is a sketch of where in the early universe the light that we'll eventually see is emitted (left), and the situation now, when we receive that light. Notice that there's nothing special about our location, every point in the universe experiences the same thing.

If the universe started as an explosion in one location, then the light from it would be long gone, and if we looked way out into space (and far back in time) all we would see is an extremely empty ancient universe.  But again, that’s not what we see.

At about the same time that Hubble and his bosom buddies in the theoretical cosmology community were contemplating this stuff a couple of astronomers testing out a new radio telescope (“the Horn”) found themselves saddled with intractable equipment problems. They wanted to use radio waves to look at stars, but before they started they decided to zero-out their equipment by pointing the Horn at the emptiest patch of sky they could find. However, they soon found that the entire sky has a uniform radio hum that was completely independent of direction. This is now called the “Cosmic Microwave Background” (CMB), and it’s the oldest stuff in the universe (other than everything else).

"The Horn" is the radio telescope that first (accidentally) detected the microwave background. To this day no one knows why it's called the Horn. Some mysteries are just too big.

Originally it was thermal radiation from the extremely hot early universe. “Hot”, by the way, is an understatement. Imagine a fire that isn’t red or white hot, but “gamma ray hot”. As in, the light it produces would irradiate you. However, because of the metric expansion it has since been red-shifted (a lot) and now has such a low frequency that it’s barely noticeable.

Light (being a wave, that's bigger than a point) gets stretched out by the expansion of space. What started out as very high-frequency, short-wavelength light has become low-frequency, long-wavelength light (specifically: microwaves).

If you tune your radio to an empty channel a substantial fraction of the static you’ll hear is the CMB (especially at night, the sun is “noisy”).

Unfortunately, when matter gets hot enough it becomes ionized, and ionized stuff is good at scattering light (that’s why you can bounce radio transmissions off of the ionosphere).  So, the CMB, the farthest back we can physically see, is from the “photon decoupling event“, when the universe cooled enough for the matter to become non-ionized and allow light to pass safely through.  This happened around 400,000 years after the big bang, which seems like a lot, but it’s only about 0.003% of the universe’s current age.

Most of the other evidence for the big bang is a bit more complicated and circuitous. For example, cosmological computer models that describe how galaxies form and move tend to work very well using the big bang as a starting point, and tend to work not even a little otherwise.

With very careful analysis (using general relativity and detailed observations of the universe today) we can talk about things even farther back than the photon decoupling event, down to within a second of the big bang itself.

However, at time zero it’s hard to say much of anything with any kind of certainty.  The closer you get to the first moments of the universe, the louder and more cantankerous physicists become.  You’d be hard pressed to find a cosmologist who disagrees that the big bang happened, and just as hard pressed to find two who agree on all the details.