Why the shortest route looks curved on a chart
A great circle is any circle on the surface of a sphere whose centre coincides with the centre of the sphere. The equator is one; every meridian pair is one; the shortest path between any two points is an arc of the one that passes through them. On a Mercator chart, where meridians are drawn parallel, that arc appears as a curve bulging toward the nearer pole — which is why transatlantic tracks look as if they detour north when they are in fact going the direct way.
The saving is real and large. A rhumb line — a track of constant compass course — from Los Angeles to New York is meaningfully longer than the great circle, and on genuinely long east-west routes at high latitude the difference runs to hundreds of miles. Below about 300 NM the two are within a mile of each other and nobody bothers with the distinction.
The price of the great circle is that the true course changes continuously. On the default Los Angeles to New York pair the initial bearing is 066° and the final bearing 094°: you set off north of east and arrive south of east, having turned 28° without ever changing your track. That is why long routes are flown as a sequence of legs between waypoints, each flown on a constant heading, rather than as one course.
The same formula serves marine navigation, where great-circle sailing is the standard method for ocean crossings, and it serves any application that needs a surface distance from coordinates — range rings, search patterns, radio coverage, freight rating.
The haversine formula and why it is used
The distance depends only on the central angle c subtended at the Earth's centre: d = R · c with c in radians. Everything else is finding c.
The direct route is the spherical law of cosines, cos c = sin φ₁ sin φ₂ + cos φ₁ cos φ₂ cos Δλ. It is correct, and for short distances it is numerically terrible: when c is small, cos c is very close to 1 and taking the arccosine amplifies rounding error dramatically. On single-precision hardware the classic failure was distances under about a kilometre coming out wrong or zero.
The haversine form avoids that by working with the half-angle sine, which is well-conditioned near zero: a = sin²(Δφ/2) + cos φ₁ cos φ₂ sin²(Δλ/2), then c = 2·asin(√a). The name comes from the haversine function, hav θ = sin²(θ/2), tabulated for navigators long before computers precisely because it made this calculation tractable by hand.
Bearings come from a different relation. The initial bearing is atan2(sin Δλ · cos φ₂, cos φ₁ sin φ₂ − sin φ₁ cos φ₂ cos Δλ), and the two-argument arctangent is essential: it resolves the quadrant correctly, which a plain arctangent cannot. The final bearing is obtained by computing the initial bearing of the reverse route and adding 180°.
All of this assumes a sphere. The Earth is an oblate spheroid, flattened by about one part in 298, so a spherical distance can differ from the true geodesic by up to roughly 0.3% — about 6 NM in 2,000. Where that matters, the standard solution is Vincenty's iterative method on the WGS84 ellipsoid.
Worked example: Los Angeles to New York Kennedy
LAX is at 33.9425°N, 118.4081°W and JFK is at 40.6398°N, 73.7789°W. Work in radians throughout.
- Convert. φ₁ = 0.592411, φ₂ = 0.709304, λ₁ = −2.066626, λ₂ = −1.287699 radians.
- Differences. Δφ = 0.116894 and Δλ = 0.778927 radians.
- First haversine term. sin(Δφ/2) = sin(0.058447) = 0.058414, and squared that is 0.0034121.
- Second term. sin(Δλ/2) = sin(0.389464) = 0.379695, squared 0.144168. cos φ₁ = 0.829599 and cos φ₂ = 0.758819, whose product is 0.629520. Multiply: 0.629520 × 0.144168 = 0.0907560.
- Sum and solve. a = 0.0034121 + 0.0907560 = 0.0941681. √a = 0.306868, asin gives 0.311900, so c = 0.623800 radians, which is 35.741°.
- Distance. d = 6371.0088 × 0.623800 = 3,974 km, which is 3,974 ÷ 1.852 = 2,146 NM or 2,469 statute miles.
- Initial bearing. y = sin Δλ · cos φ₂ = 0.702512 × 0.758819 = 0.533078. x = cos φ₁ sin φ₂ − sin φ₁ cos φ₂ cos Δλ = 0.540319 − 0.301537 = 0.238782. atan2(0.533078, 0.238782) = 065.9° true.
The final bearing works out to 093.9°, so the track swings 28° right across the continent. At a typical airline ground speed of 480 kt the leg takes 2,146 ÷ 480 = 4 h 28 m in still air.
Using the distance and the bearings
The distance is the input to your timing and fuel plan. Take it to the flight time and ETA calculator with a planned ground speed, then to the fuel burn calculator. Remember that it is the geometric distance: your actual track miles will be longer once departure procedures, airways, terrain avoidance and arrival routings are added, commonly by 3-8% on a domestic route.
The initial bearing is a true bearing. To fly or steer it, apply magnetic variation for your position — and note that on a long route the variation itself changes along the track, which is a second reason to break the route into legs.
The difference between the initial and final bearings tells you how curved the route is in course terms. On a north-south route it is nearly zero. On an east-west route at high latitude it can exceed 60°, and there is no sensible way to fly it as a single heading.
The midpoint is the equal-time point only in still air. With wind, the equal-time point — the position from which continuing and turning back take the same time — moves toward the upwind end, sometimes a long way. For a genuine ETP calculation you need both the distance and the wind, which means working the ground speed in both directions.
The waypoint table in the results gives you the track at eighths of the way along, which is enough resolution to plot the route on a chart or to check that it does not cross terrain, restricted airspace or, on an ocean crossing, a landmass you had not considered.
How far is one degree?
| Latitude | cos φ | 1° of longitude (NM) | 1° of longitude (km) |
|---|---|---|---|
| 0° (equator) | 1.000 | 60.04 | 111.20 |
| 10° | 0.985 | 59.13 | 109.51 |
| 20° | 0.940 | 56.42 | 104.48 |
| 30° | 0.866 | 52.00 | 96.30 |
| 40° | 0.766 | 46.00 | 85.17 |
| 50° | 0.643 | 38.59 | 71.47 |
| 60° | 0.500 | 30.02 | 55.60 |
| 70° | 0.342 | 20.54 | 38.03 |
| 80° | 0.174 | 10.43 | 19.31 |
One degree of latitude is 60.04 NM on this sphere and exactly 60 NM on the nautical sphere option, which is the historical definition of the nautical mile: one minute of arc.
Traps in coordinate distance calculations
- Sign errors on longitude. West is negative and east is positive here. Entering 118.4081 instead of −118.4081 for Los Angeles puts the point in central Asia and produces a distance of the wrong hemisphere entirely.
- Degrees-minutes-seconds entered as decimals. 34°30′ is 34.5, not 34.30. That single error is worth 18 NM of latitude.
- Using the great circle as the flight-planned distance. Airway routings, SIDs and STARs add track miles. Use the great circle for a floor, not for the fuel plan.
- Flying the initial bearing all the way. On a long route that is a rhumb line, not a great circle, and it lands you off track. Break into legs.
- Forgetting magnetic variation. All the bearings here are true. On a long route the variation changes along the track as well.
- Assuming spherical equals geodesic. The ellipsoidal answer can differ by up to about 0.3%. For legal, surveying or precise range work, use Vincenty or a geodetic library.
- Treating the midpoint as the point of no return. Wind moves the equal-time point, often substantially, and fuel state moves the point of no return further still.
Great circle, rhumb line and geodesic
Three different curves get used for the same job. The great circle is the shortest path on a sphere and what this page computes. The rhumb line, or loxodrome, holds a constant true course, so it is trivially easy to steer and appears as a straight line on a Mercator chart; it is longer, and near the poles it spirals rather than terminating. The geodesic is the shortest path on the actual ellipsoid, computed by Vincenty's method or by Karney's more robust modern algorithm, and it is what surveying and legal boundary work require.
In practice, aviation and marine navigation use great-circle distances for planning and a sequence of rhumb-line legs for execution, choosing waypoint spacing so that the cumulative departure from the great circle stays small. Modern flight management systems and chartplotters do the whole thing continuously and the distinction disappears from the cockpit — but it reappears the moment you plan by hand or check a machine's answer.
Once you have the distance, the rest of the plan follows: convert indicated to true airspeed, apply the wind with the wind correction angle calculator to get ground speed and heading for each leg, then compute time en route and fuel. For the arrival, the top of descent calculator works backwards from the destination.
Distances computed here are for planning. Published airway and route distances, and the figures in an operator's navigation database, govern operational use.