When an object is selected in Maya, a selection ray is generated based on the orientation of the camera and the mouse position. Selection happens in two steps—first, all of the object bounding boxes are tested for ray intersection, then the select functions are called for all objects whose bounding boxes were hit.
The shape selection function is defined by overriding the select function of MPxSurfaceShapeUI. You must override this function to add items to the given selection list based on the selection state information provided by MSelectInfo.
See the method apiMeshShapeUI::select and apiMeshShapeUI::selectVertices in the apiMeshShape example plug-in for an example of object and component selection.