Spring Freeze or Heat Damage
Citrus trees can drop young green fruit in the spring when a freeze hits. This can happen immediately after the freeze or be delayed. When delayed it is a function of cold damage done to the tree's woody tissues that then manifests itself systemically causing fruit drop. A sudden increase to warm temperatures around the time the fruit sets can also shock the tree resulting in the drop of tiny green fruit.
Normal Growth & Drop Cycles
Citrus trees have a self regulating mechanism that calibrates the amount of fruit the tree carries on its branches each season. As a part of this process some young green fruit can be shed. Most citrus trees cycle through two phases when you will see fruit drop. The first is a few weeks after bloom finishes when tiny green olive size fruit come down.
The second drop is in late May when larger ping-pong size fruit will be shed. When some green fruit drops in this pattern it is considered part of the normal development process.
A range of cultural problems or activities can contribute to green fruit drop beyond what is normal for citrus. Inadequate water or fertilizer or poor soil nutrition can lead to drop. Insect or pest activity can cause unusual, off cycle fruit drop. Overly aggressive pruning can trigger shock which will spur fruit drop. Too much water, particularly with container grown trees, can also induce drop early in the fruit development cycle.