This paper demonstrates an algorithmic framework for autonomous, distributed navigation and timekeeping for spacecraft swarms and constellations using angles-only measurements from onboard cameras. Angles-only methods are compelling as they reduce reliance on external measurement sources. However, prior flight demonstrations have faced limitations, including 1) inability to treat multi-agent space systems including multiple observers and targets in an accurate and timely manner, 2) lack of autonomy and reliance on external state information, and 3) treatment of primarily Earth-orbiting scenarios. The Absolute and Relative Trajectory Measurement System (ARTMS) discussed in this paper overcomes these challenges to enable future lunar missions. It consists of three novel algorithms: 1) Image Processing, which tracks and identifies targets in images and computes their bearing angles; 2) Batch Orbit Determination, which computes a swarm state initialization from angles-only measurements; and 3) Sequential Orbit Determination, which uses an unscented Kalman filter to refine the swarm state, seamlessly fusing measurements from multiple observers to achieve the necessary robustness and autonomy. This paper augments ARTMS for lunar navigation and its theoretical performance is investigated through a quantitative observability analysis. High-fidelity simulations with a star tracker in the loop demonstrate successful navigation of swarms and constellations in low lunar orbits, near-rectilinear halo orbits, and elliptic frozen orbits. ARTMS achieves absolute orbit estimation for all swarm members using only inter-satellite angles with simultaneous estimation of differential clock offsets and ballistic coefficients. It therefore presents an important capability for the support of future lunar and planetary exploration.