Anything from something that was once alive -- including nuts -- can be composted. It's a matter of how you compost, how much time you have to compost the material, and where your compost is designed to end up. There's also a difference between types of nuts and whether you're composting the insides or the shell.
Nut shells often take longer to break down in a compost pile than other compostable materials. The shells are considered "brown," or carbon-rich materials, so they help balance out "green," or nitrogen-rich materials in the compost pile. If you intend to compost that particular pile for a long time, nut shells will get a chance to decompose. Nut shells are also a good bulking agent, as they create pockets of air and keep the pile from getting so dense that air cannot circulate. A lack of air circulation makes a pile decompose incorrectly and smell bad.
Worm composting rarely breaks down nut shells. Worm composting, or vermicomposting, relies primarily on red wiggler earthworms to break down household scraps into fine compost. Pistachio nut shells often break down, but heavily salted nuts in a worm bin will adversely affect the worms. Nut meats break down fine in most any compost system, including worm composting.
Walnuts contain compounds that inhibit the growth of some other plants, such as tomatoes. The concern about using composted walnuts in the garden is that some of the growth inhibiting compounds may not have broken down, especially if pieces of walnut shell are still intact. However, the Cornell University notes that woody parts and husks of walnuts composted for six months should be safe even for sensitive plants. When in doubt, avoid composting walnuts or add composted walnuts to areas such as natural walkways where you don't want weeds to spring up.
Crushing the shells of nuts before adding them to the pile helps them decompose faster. It also ensures that, even if the shells don't break down, the compost isn't so coarse as to be unusable. If you're using this compost as mulch or in other areas that benefit from coarse compost, bits of nut shell in the finished product should not be problematic. If you need fine compost, such as for seed planting projects, shaking the compost through a screen will filter out the larger bits of shell.