Use fine gravel where a really solid base is required and subsurface drainage is not a concern. Fine gravel compacts almost like sand into a really tight surface. That may not provide enough drainage or seepage to allow water to permeate into the ground below, especially if it is tight or heavy dirt like clay. That can lead to pockets of water under the pavers, which freeze and cause the paving to heave or buckle.
Medium gravel is the choice for most paver bases. It compacts tightly, but with small gaps between stones so water can percolate into the dirt below. It forms a base solid enough to hold a sand bed for the pavers and strong enough for most weights put on pavers, even standard cars in a driveway or big smokers on a patio.
Use coarse gravel where the pavers and base will be subject to heavy weights, such as a driveway where multiple vehicles, bigger trucks and heavy boats will be driven and parked. Coarse gravel will not compact as tightly, but the larger stones provide more support under heavy loads. Coarse gravel is sometimes used as a base course or layer and topped with medium gravel before a sand bed is added. Coarse gravel allows good subsurface drainage.
Consider repurposed gravel, old concrete that has been crushed just like limestone (limestone aggregate is a component of most concrete). Recycled concrete gravel can work as well as limestone in most applications. It is usually cheaper and always will conserve natural limestone resources. Lay it and compact it just like limestone gravel.