Why are optical disc drives slower than hard disk drives?

There are two separate questions: latency and throughput.

Seek time

Optical drives have random seek times around 100 ms, whereas hard drives are around 10 ms - why is this?

An optical read/write head consists of a laser, detector, mirrors, lens, and multiple voice coils to position the head/lens for tracking/focusing. This entire assembly has a number of parts and is relatively heavy. The entire head assembly moves on a worm gear.

A hard disk head consists of a tiny GMR sensor for reading and coil for writing. The rest of it is the plastic slider and metal arm; the whole assembly is relatively light. The head is moved by a powerful voice coil. This is why seeking is fast.

Transfer rate

CD drives top out at about 10 MB/s. DVD drives top out at about 30 MB/s. Blu-ray drives top out at about 70 MB/s. Hard drives routinely hit 100~200 MB/s.

The two main factors that determine transfer rate are linear speed (metres per second) and linear density (bits per meter).

Desktop hard drives in the 3.5-inch form factor have a platter diameter of about 90 mm. Optical discs have a diameter of about 120 mm. I would rate these as "close enough" for comparison purposes.

Hard drives usually spin at 5400 RPM or 7200 RPM, with old enterprise models going up to 15000 RPM. Optical discs are spun at various speeds, depending on how quickly the host wants to read/write data (e.g. bulk read vs. streaming audio/video), how long the drive has been actively used, how much noise is desired, etc. But optical discs can spin up to about 10000 RPM in real drives without problems. So this is also in the same ballpark as HDD RPM.

Areal density is a huge factor to consider. Pretend for a moment that a CD is 1 GB and an HDD is 1000 GB and they have the same physical dimensions. Clearly, the HDD has 1000× the data density per area. As for linear density, the HDD is √1000 = 32 times denser than the CD. So if you position a head over the discs and make one full revolution, the HDD should read 32× more data than the CD, simply because more data is packed on each track. As we can see, this is why DVDs and BDs have higher transfer rates than CDs. But a 25-GB single-layer Blu-ray disc absolutely pales in comparison to even a cheap, basic 1 TB HDD. Though however, multi-layer optical discs and multi-side multi-platter hard disks make this calculation more complicated.

Bonus

To add insult to injury, my experience shows that after inserting an optical disc into a drive, it takes about 20 seconds to begin reading any user data on it. This start-up time is far worse than flash drives and even hard drives, and is especially painful when sifting through many discs.

While the other answers are correct, there is more. Optical media is manufactured to be cheap and lightweight. Its just a small piece of plastic. It it not designed to spin at high speeds. The faster it spins, imperfections in the plastic start to cause the plastic to warp. The faster it spins, the more it warps. This warping will make it unreadable by the drive. Spin the media fast enough and it will fail... violently. Here is a video showing the warping and failing at the extreme.

The fastest CD/DVD/BD optical drives spin at ~10k RPM at most.
Really high-end magnetic drives spin at 15k RPM, although consumer HDDs (and bulk-storage enterprise HDDs) typically spin at 7200 or 5400 RPM.

Hard drives are precision assembled in a clean environment and sealed. They can be (and are) made to very high tolerances.

CD drives are open and designed to accept media that are clean but may have dust and fingerprints on the surface. These cannot be built to the same tolerances as sealed hard drives.

CD drives turn slowly. Hard Drives turn at 7200 rpm (and some expensive drives at 15,000 rpm) in order to provide faster data access and transfer. Tolerances have to be different. The HDD head moves over the platter surface in as little as 3 nanometers. Dust (cannot be introduced except by opening the drive) or stopping the drive rotation while the head was over the drive would damage the platter. Particulate matter can be created in a crash.

They are very different devices used for very different purposes and are not designed to be the same.

Don't forget, the CD-ROM was an extension of the audio CD. The underlying technology was originally designed with music in mind. It was designed to be fast enough for audio playback, and then later Sony piggybacked the CD-ROM on top of it. This isn't a format designed with high-speed data transfer as the target use case.

But why do HDDs spin faster

Because rotational speed (and performance in general) is a design priority for hard drives. Hard drives are precision-manufactured devices. The spinning platters are almost perfectly uniform in terms of material composition, and are permanently attached to the motor shaft.

Optical drives, on the other hand, were designed for a business case where cost is more important than pure performance. They're primarily made out of fiberglass because it's cheap, transparent, and lightweight. An optical disc's density is far from uniform, however. That means they're not as perfectly balanced as the platter in a hard drive. When you spin something that's not balanced, it wobbles. Vibration is an enormous problem in optical drives since the read mechanism involves a laser reflecting off the disc surface; a vibrating disc sends reflections off at an angle, missing the detector and making it more likely that a bit will be misread. That's a big reason why drive read speeds haven't gotten much faster in decades. Ever used a portable CD player and jostled it enough that it skipped? That's what's happening. DVD and Blu-Ray drives actually spin slower than a lot of CD-ROM drives precisely to minimize vibration-related problems.

Optical discs are also removable, which means they only attach to the motor's shaft via a friction-fit system. If the shaft spins too fast or accelerates too quickly, its grip on the disc will slip and the disc and shaft will start grinding on each other.

Why don't they manufacture the arm inside the optical disc drives to move at the speed of HDDs?

The read heads of these two drives use wildly different mechanical principles. The read head on a hard drive looks somewhat like a record player. It's a long arm dangling over the spinning platter that can pivot to reach different points along the disc's radius. The arm does not contact the platter, and can whip back and forth through the air relatively quickly thanks to some strong electromagnets.

An optical disk uses a laser mounted on a rail. A motor pulls a chain that makes the laser module slide back and forth across the rail. This type of mechanism has significantly more mass and friction than a hard drive's air-suspended read head. There's no reasonable way to make it move as quickly without damaging something.

Also, can't the head go to a specific part of the spiral track and read from there?

Absolutely, that's how you can seek to the beginning of a specific song on a CD.