Use a hose or rope to mark a circle around the stump; the circumference of the circle should be 1 1/2 to 2 feet wider than the circumference of the bush that used to inhabit the space.
Use a shovel to dig a trench around the line you've marked, then move the hose or rope out of the way.
Dig from the outside edge of the circle in toward the stump, removing the soil to a wheelbarrow or making a pile of soil outside of the circle. How deep you dig will depend on the size of the stump to be removed; plan on 1 foot at a minimum and up to 3 or 4 feet for larger stumps.
Use an axe or hatchet to chop through any thick roots that extend beyond the circle or go deeper into the ground than you are shoveling.
Use the shovel as a lever to lift the stump from the roots up: push the shovel blade as far under the stump as you can, then push down on the handle to lift the stump up. You may find additional roots holding the stump down that need to be chopped off.
Backfill the hole with the dirt you removed once you have the stump out; tamp down firmly to prevent the area from sinking, then cover with mulch or sod to keep the soil from eroding.