Part 1: Using your Scanner

Your scanner is an integral part of your gameplay experience, if a bit complex and hard to get the hang of. It's a pirate's single most useful tool, and a non-pirate's best protection (aside perhaps from always keeping an eye on one's Local chat channel to see who's in the system).

You can open the scanner using the scanner button (one of the little buttons to the left of your ship's information while you're in space). It's a good idea to practice plenty ahead of time, and to get a Scanner Settings selection saved to your Overview Settings well before you might need to use it in a sticky situation.

When you open your scanner in space, there's a small box you'll want to tick. This is the "Use Overview Settings" box. What it means is that the scanner will only show the things you have shown on your Overview. In this way, you can avoid showing a multitude of Concord ships, or sentry guns, or Concord Billboards, or whatever isn't important to you.

The second step is to enter, in the box showing distance in kilometers (km), as many 9's as you can fit. Hit enter a couple of times, and it will reset to the default scannable distance. Now you'll be scanning as far as you can scan; 150 million km is about equal to one AU, for reference.
Next, hit Scan. Right-click everything you don't want to see in there, everything that isn't important to you, and pick Remove from Overview. Once you're done, and the overview settings are to your liking, go into your Overview (on the far right, by clicking on the small white arrow beside it) and go to Save Current Type Selection As, and name it Scanner Settings or the like. This way, you can just switch your overview to this and scan when you need to see what's out there.

Note: Before you begin using your scanner in precarious situations, you'll want to do two things: first, practice a lot. Second, and perhaps just as important, if you're going to be scanning (whether for predators or prey) in low-security space, make at least a basic safespot in the system and do it there. It's best, too, if you are moving constantly, rather than sitting still--this will make it much harder for enemy ships to jump on top of you. If you're sitting in an asteroid belt scanning at the warp-in point, chances are you won't last very long. To make a basic safespot, just open up your People & Places tab, warp from one section of a system to another, and drop a bookmark by hitting Add Bookmark and immediately hitting Enter. You can change the name later by right-clicking it and picking Edit/View on it. It's best if the safespot is between two other safespots, rather than an "inline" one (in a line between two warpable objects, i.e. a jump gate and a planet); this too will make you harder to find. Moving often between safespots will also help keep you safe. For scanning purposes, you may want to make a spot either just outside the system's "inner clump" of star and closely-orbiting planets so you can narrow down your scanner beam, or you might want to sit closer to the inner clump itself so you can see everything around you. If you see someone in Local chat that you can't find, press F11 to bring up the system map (bottom right of the screen), which will (once you change scan degree range, say from 360 to 180) show you where you are searching. Bringing up this system map can show you if there's a group of planets far out on the edges of the system; this may mean the person you're looking for is out there somewhere--otherwise, they may be docked or cloaked.

  The System Map

As has been mentioned, pressing F11 will bring up the system map. Change the degree selection on your scanner and you'll see a green cone on the system; this shows the area you're scanning. It's limited to one-hundred-eighty degrees, however. Keeping F11 pressed will, unfortunately, crowd your screen, so try not to go into a fight with it still on! The system map is also limited by the fact that, when you are in the central grouping of planet, moons, and sun in a system, it can get difficult to narrow down your search using it, since too many things are within your scan cone. In this case, press F11 again to get rid of the map, and search by eye. To do this, just narrow your search beam (you may need a lot of practice to figure out where the 'center' of your screen is compared to the scanner) and press and hold Alt. This will bring up the icons of planets, moons, and docks, regardless of what your overview settings are. Note: this is a good reason not to bind "Alt" to your push-to-talk voice command.

A Note About Moons

Scanning from moons while in low-security space is, in general, a bad idea. It may make you slightly safer than scanning from a planet, since it's harder for enemies to narrow you down on the scanner, and because they're reluctant to go there unless they know it's safe. But the reason people are less likely to follow you to a moon is simple: Moons in low-security space often harbor POS's--Player Owned Structures. These consist of several things, but the important one to consider is guns. If it looks like a ship is at a moon (say, Joe Smith's Caracal) and the owner isn't in system, it's a safe bet that the ship is at a POS. POS's are run by a control tower inside a large forcefield, and usually are set to kill anyone who gets too close. You can add Control Towers to your overview by going into your Overview Settings and going to filters/types/structures, then checking the Control Towers box. In general, though, you'll want to avoid moons in low-sec if you can help it!

  Narrowing Down Targets

If you see someone "on the three-sixty," meaning their ship shows up on your scanner when it's set to three-hundred sixty degrees, you can narrow them down and try to pinpoint their location. First, drop it to 180. If they don't show up, swivel your ship view around to the opposite side and try again. If they appear again, narrow the cone of scanning down to 90. Turn the view slightly to the left and right until you see them again. Once you have them narrowed down to 90, try pressing Alt in space. If there's only one object in that radius that one can warp to, narrow the beam (or camera, if you're not using F11) down to 30', with that object dead in the center of your screen. You can narrow it further once you get more practice, too. If they still appear on your scanner, it's a safe bet they're there. If they're not, they may be at their own safespot, and then you'll need scanprobes to find them (more on that soon).
If there are multiple warpable items within your 90 range, you can narrow the beam further and try to get the target and a warpable object together in the beam, so that you know where they are. If there are too many things to warp to in that range, try heading in that direction and scanning from there--for example, if you see Planet V, Planet VI, all their moons, plus two stations over that way, you may want to warp (not to 0 in lowsec, to be safe) to one of the planets and scan from there to see if you can pinpoint them.

 
Part 2: ScanProbes

Scan Probes were present in Eve before the Revelations patch, but have been drastically changed since then. In short, scan probe launchers launch scan probes, which when launched allow you to probe for, and warp to, ships or hidden objects in space. Eve-Online itself has an excellent guide here. Very basically, if you intend to scan for enemy ships, you'll want to do two things: buy a Recon Probe Launcher I (which will drastically reduce scanprobe times), or a Scan Probe Launcher if you're looking to use deadspace probes, and train up the relevant skills (such as Astrometrics, Signal Acquisition and Astrometric Triangulation).
Then you need to select the scan probes you'll be working with. If you want to search for hidden asteroid belts and hidden deadspace complexes, you will want to use the "Quest" probes. If you are searching for ships, though, use the various ship-scanning probes, such as the Observator Deep Space Probe, or the Fathom, Snoop or Spook probes. Each varies according to how long it takes to scan--the longer it takes, the more range it has, but when you warp to the object in space, a longer-range probe will tend to pinpoint a little less accurately. If you have no idea where someone is in a system*, you'd start off with a long-range probe, then warp to that spot, and if the person isn't visible, try a shorter-range probe. Once you're done with probes, it's a good idea to open up your scanner and right-click the probe, and select Destroy Probe.

To actually use the probe launcher in space, you'll need to equip it to your ship. Then load it with scanprobes, and take along (in your cargo hold) extra probes of whatever types you think you'll need. Then activate the scan probe module (it works similarly to a weapon turret) and it will launch a probe; it's best if you are pretty sure there's a target in the scanprobe's range. Once you've done this, open your Scanner. Click on the far left tab, select the scan group you want, then select the scan probe and click Analyze. A timer will count down. Once it's done, you'll see a results sheet; this will show Accuracy (range in kilometers or meters that tells you how far you might end up from your target; short-range probes are far more accurate), distance, and item name/type. You can then right-click on the object and select Warp To, and the distance.

Note: If you're in a CovertOps ship that's warping cloaked, and you want to STAY cloaked, be sure to not warp directly onto your target. So if the accuracy is 0m, warp to 10 or 20 km rather than 0m! Another note: if you're having trouble with scan probes taking so long to analyze that they expire before the scan is done, make sure you get a Recon Probe Launcher, and train up Signal Acquisition.

This is just a basic introduction to scanning and scan probing. Read that Eve guide linked above, and practice!

* If you start out with a long-range probe because you don't know where a person is, make sure there are no POSs (as described above) in the system... otherwise you could warp onto a ship safely secured within the POS, which will then destroy your ship!