All these books were personally reviewed and are recommended by Hershel Friedman, the author of this site.
 

National Audubon Society Field Guide to North American Rocks and Minerals
By Charles W. Chesterman
This book is, in my opinion, the greatest field guide for mineral collectors. Its mineral descriptions are very detailed and informative, and its content is geared for scientifically oriented as well as casual collector. This book has over 700 color photographs.

A Field Guide to Rocks and Minerals (Peterson Field Guides)
By Frederick H. Pough
This is not really a field guide; it is more like a detailed and informative guide to minerals. This is one of my favorite sources for information, as its accuracy and comprehensiveness are unparalleled. The later editions of this book also have excellent pictures taken by reknowned photographer Jeffrey Scovil.

Gemstones of the World
By Walter Schumann
The most complete guide to gemstones and jewelry I have seen. This book discusses in detail all gemstones, including rare collectors gemstones. The pictures are equally notable, with both rough crystals and cut gems. This is the number one book on gemstone information.

Eyewitness Handbooks: Gemstones
By Cally Hall
This book is a handbook on gemstones and jewelry. The pictures are exceptional, and the is perfect for students and those who want the basic information. Although not nearly as detailed as "Gemstones of the World", this book still provides all necessary information.

The World of Opals
By Allen W. Eckert
This book is no doubt the most exhaustive and complete reference for opal. It provides detailed gem information as well as technical opal specifications. A must for all opal hobbyists.