A convex cone K that is a subset of the positive semidefinite (PSD) cone is rank one generated (ROG) if all of its extreme rays are generated by rank 1 matrices. ROG property is closely related to the characterizations of exactness of SDP relaxations, e.g., of nonconvex quadratic programs. We consider the case when K is obtained as the intersection of the PSD cone with finitely many linear (or conic) matrix inequalities, and identify sufficient conditions that guarantee that K is ROG. In the case of two linear matrix inequalities, we also establish the necessity of our sufficient condition.