Layers > Delete > Unused Layers
 
 
 

Deletes all layers with no objects on them.