The root systems of trees are typically at least 1½ times as large as the canopy of the tree they support. The size and structure of the roots mimic the size and structure of the leaf canopy as well. In order to thoroughly remove the root system, you'll need several different cutting tools and a freshly sharpened garden spade.
A sharp spade will help to remove tree roots in two ways. First of all, it will dig down into the soil to locate them. Second, if your spade is sharp enough it will sever the roots, making it easier to remove them. Small pieces can usually be removed much easier than larger ones, particularly if the ground is packed and hard.
An ax may be necessary to hack away at unyielding ground or large, branch-size tree roots. This is especially helpful if the roots have grown around a rock or other immovable object in the soil. Use the ax to break off the root so you can remove any parts of it that are not hemmed in by these underground objects.
Pruning Clippers or Saw
Pruning clippers or a pruning saw can be used to cut small to medium size roots when using an ax would be overkill. If the roots to be removed travel into the root system of a plant that you don't want to dig up, use the clippers or saw to cut it off just outside the root ball of that plant.