I ran into this problem when I want to simulate subsatellite points.
There are generally two kinds of descriptions or definitions of the subsatellite point:
Wikipedia even gives a contradict description (https://en.wikipedia.org/wiki/Ground_track):
A satellite ground track may be thought of as a path along the Earth's surface which traces the movement of an imaginary line between the satellite and the center of the Earth. (This obviously use the first definition.) In other words, the ground track is the set of points at which the satellite will pass directly overhead, or cross the zenith, in the frame of reference of a ground observer. (but this obviously relies on “zenith”, the second definition above.)
https://space.stackexchange.com/a/19729/28220
For Earth observation satellites, “nadir” points toward the geodetic subsatellite point. The geodetic subsatellite point is that point on the surface of the reference ellipsoid that is closest to the satellite. This is equivalent to the point on the ellipsoid where the satellite is at zenith (“straight up”). Using nadir as pointing to the center of the Earth will miss geodetic subsatellite point by up to 21 km for a satellite in a sun synchronous orbit. (I should use this one for ground track simulation.)
For satellites formation flying, and rendezvous and proximity operations, “nadir” and zenith point to / away from the center of the Earth, in the LVLH frame. This is primarily because of the use of the linearized Clohessy-Wiltshire equations in these fields.
An even more subtle meaning is that nadir points in the direction of the point on the geoid that is closest to the satellite. (see original post. I don't understand)