Associated Topics || Dr. Math Home || Search Dr. Math

### Calculating the Shortest Flight Path

```
Date: 10/30/97 at 20:48:26
From: Phil
Subject: Calculate angle between two places

Using flight simulator, flying long distance between two points with
given latitude/longitude, calculate the flight path (angle) which is
the shortest route.

Is this spherical trigonometry or can it be solved using simpler
geometric equations?
```

```
Date: 10/31/97 at 08:27:38
From: Doctor Jerry
Subject: Re: Calculate angle between two places

The bearing of the great circle path (shortest) at the initial point
can be calculated, as you suspect, by spherical trigonometry. The word
"simpler" is tricky, however. For me, knowing almost no spherical
trigonometry, it's simpler to use spherical coordinates, vectors, and
dot products. This gives a nice method for calculating the bearing.

If you are interested, please write back.

-Doctor Jerry,  The Math Forum
Check out our web site!  http://mathforum.org/dr.math/
```

```
Date: 10/31/97 at 08:52:34
From: Phil
Subject: Re: Calculate angle between two places

Thanks for replying. Yes, I would like to know how to calculate the
shortest path (angle) between two sets of co-ordinates (lat/long)
using the method you mentioned.

Thanks in anticipation.
```

```
Date: 10/31/97 at 15:26:02
From: Doctor Jerry
Subject: Re: Calculate angle between two places

Hi Phil,

It's easiest for me to explain in terms of an example. I'll use
spherical coordinates, which are closely related to the geographic
coordinates of latitude and longitude, and I'll work on a sphere with
radius 1. I think the solution procedure will be clear for the general
case. I don't know that the solution I'll explain is the simplest. If
one were to read a book written for navigators, I presume that
relatively simple ways to calculate the bearing of a great circle
route would be explained. However, from my experience in looking at
such books, to understand the procedures I would have to read more of
the book than I'd want.

I'll consider points S (starting point) and E (ending point);
specifically, S has spherical coordinates (1,pi/4,pi/6) and E has
spherical coordinates (1,pi/6,pi/3). These correspond to (on the unit
sphere) points with latitudes 45 deg N and 60 deg N and longitudes
differing by 30 deg.

I'll use the spherical to rectangular transformation

r(phi,theta)={r*sin(phi)*cos(theta),r*sin(phi)*sin(theta),r*cos(phi)}.

Start by calculating the rectangular coordinates of S and E; call
these P1 = r(pi/4,pi/6) and P2 = r(pi/6,pi/3). We may think about P1
and P2 as vectors from the origin.  From these two vectors we form
mutually perpendicular vectors u and v lying in the plane containing
P1 and P2. The intersection of this plane with the sphere is the great
circle of the shortest route. I'll explain in terms of the Mathematica
code I used to check all of this. The code is a little peculiar, but I
think you can follow it with no difficulty.

r[phi_,theta_]:={Sin[phi]*Cos[theta],Sin[phi]*Sin[theta],Cos[phi]}

The curly brackets make r[phi,theta] a vector.

Let P1 = r[Pi/4,Pi/6] and P2 = r[Pi/6,Pi/3].

Let u = P1.

The vector w

w = P2-((P1.P2)/(P1.P1))P1

is a vector perpendicular to u = P1. The expression P1.P1 is dot
product. Lgth is the length function.

Now define v = w/Lgth[w].

The great circle route through S and E is the parametric curve

r1[t_]: = Cos[t]*u+Sin[t]*v

for 0 <= t <= 2*pi.

Note that r1[0] is the point S. To calculate the bearing I'll
calculate the tangent vector r1'[0] and dot it with the north-pointing
unit vector m through S.  Differentiating r1,

r1'(t) = -sin(t)*u+ cos(t)*v.

Hence, r1'(0) = v, which was obvious anyhow. Also,

m = {-sqrt(3)/(2*sqrt(2)), -1/(2*sqrt(2)), 1/sqrt(2)}

This vector can be calculated by forming the negative of the tangent
vector to the curve with parametric equation r(phi,pi/6), and
evaluating at phi = pi/4.  The bearing is then

b1 = ArcCos[(m.v)/(Lgth[m]*Lgth[v])]

In particular,

b1 = ArcCos[3*sqrt(3)/(8*sqrt(5))+sqrt(15)/8]=0.684719 or
39.23 degrees.

Although this procedure probably will require some special cases for
points in different hemispheres and other quirks of spherical or
geographic coordinates, it is fairly straightforward.

I checked all of this by doing it a different way. I assumed the
equation of the plane through the center of the earth, S, and E had
the form A*x+B*y+z = 0 and, with the rectangular coordinates of S
and E, calculated A and B. This gave the plane -2y+z = 0.  I solved
this simultaneously with x^2+y^2+z^2 = 1, thinking of z as fixed.

This gave me parametric equation (parameter z) for the great circle.
I found

r2[z_]: = {Sqrt[4-5z^2]/2,z/2,z}

I calculated r2'[0] and dotted it with the meridian vector as before
and got the same bearing.

-Doctor Jerry,  The Math Forum
Check out our web site!  http://mathforum.org/dr.math/
```

```
Date: 06/14/2005 at 06:28:49
From: Nick
Subject: Distance Between Points on the Earth

quite simply without using dot products, scalar products or
coordinate transformations:

Calculating the "Great Circle" distance from P1 (Longitude X1 West,
Latitude Y1 North), to P2 (X2,Y2).

(i) - The length of a "vertical" drop (vertical means parallel to the
north/south axis) from P1 to the equatorial plane is given by
RsinY1 where R is the Earth's radius.  This vertical will intersect
the equatorial plane at a point a distance RcosY1 from the Earth's
centre.  Call this point P3.

(ii) - The length of a vertical drop from Y2 to the equatorial radius
is given by RsinY2.  To a point RcosY2 from the earth's centre (P4).

(iii) - Thus we have a trapezium with three orthogonal sides:

1) - The "base" is P3/P4, and its length can be calculated since
the lengths of two sides and the angle between them are known, using
the Cosine Rule: (a^2 = b^2 + c^2 - 2bc*cosA) where: "a" is base P3/P4;
b and c are the two distances from the Earth's centre, RcosY1 and RcosY2;
angle A is the angle between them which is the difference between the
two longitudes: X2-X1.  So:

(P3/P4)^2 = (RcosY1)^2 + (RcosY2)^2 - 2(RcosY1)(RcosY2)cos(X2-X1)

2,3) - The two parallel sides of the trapezium are orthogonal to
the base (P3/P4) and are the "verticals" given in (i) and (ii) with
lengths of RsinY1 and RsinY2.

(iv) - The length of the fourth side is the straight line distance
between P1 and P2.  The trapezium is effectively a rectangle joined
with a right-angled triangle.  The "fourth side" of the trapezium is
the hypotenuse of the triangle; the base of the triangle is the same
as P3/P4 (see (iii)-1); the third side of the triangle is the
difference between the lengths of the two parallel sides of the
trapezium: RsinY1 - RsinY2.  Pythagoras gives us:

(P1/P2)^2 = (RsinY1 - RsinY2)^2 + (P3/P4)^2

(v) - Knowing the straight line length of P1/P2 enables the
calculation of the angle subtended between P1, P2 and the centre of
the Earth, using the Cosine Rule A = arccos(b^2 + c^2 - a^2)/2bc,
where b and c both = R and a = P1/P2.  So:

A = arccos({R^2+R^2-[(RsinY1-RsinY2)^2+(RcosY1)^2+(RcosY2)^2-2
(RcosY1)(RcosY2)cos(X2-X1)]}/2R^2)

Removing the R^2 terms gives:

A = arccos({2-(sinY1-sinY2)^2-(cosY1)^2-(cosY2)^2+2(cosY1)(cosY2)cos
(X2-X1)}/2)

Expanding (sinY1-sinY2)^2 gives:

A = arccos({2-(sin^2Y1+sin^2Y2-2sinY1sinY2)-cos^2Y1-cos^2Y2+2(cosY1)
(cosY2)cos(X2-X1)}/2)

Then:

A = arccos({2-(sin^2Y1+sin^2Y2-2sinY1sinY2+cos^2Y1+cos^2Y2)+2(cosY1)
(cosY2)cos(X2-X1)}/2)

Since sin^2 + cos^2 = 1, this simplifies to:

A = arccos({2-(1+1-2sinY1sinY2)+2(cosY1)(cosY2)cos(X2-X1)}/2)

And then to:

A = arccos({2sinY1sinY2+2(cosY1)(cosY2)cos(X2-X1)}/2)

And finally to:

A = arccos{sinY1sinY2+cosY1cosY2cos(X2-X1)}

(vi) - The "great circle" distance along the Earth's surface between
the two points is trivial from here, since one nautical mile is
defined as the distance travelled when moving through one minute of
arc along the Earth's circumference.  So just multiply the angle A,
in degrees, by 60.  From nautical miles, the distance can easily be
converted to miles, km or another linear measure!
```

```
Date: 06/14/2005 at 15:04:18
From: Doctor Rick
Subject: Re: Distance Between Points on the Earth

Hi, Nick.

Doctor Jerry offered a method without spherical TRIGONOMETRY, which
is a separate subject, not taught much anymore because spherical
geometry and vector algebra can do the same things and are more general
in application.

Simplicity may be in the eye of the beholder. For one who is familiar
with cartesian-spherical coordinate transformations and vector
algebra, the method shown in the archives is much simpler. However,
it is surely comforting for those who do not understand these methods
to see that they are not required. In fact, much of your work is a re-
derivation and confirmation of the formulas for the coordinate
transformations and vector algebra.

- Doctor Rick, The Math Forum
Check out our web site!  http://mathforum.org/dr.math/

```
Associated Topics:
College Calculus
High School Calculus

Search the Dr. Math Library:

 Find items containing (put spaces between keywords):   Click only once for faster results: [ Choose "whole words" when searching for a word like age.] all keywords, in any order at least one, that exact phrase parts of words whole words

Submit your own question to Dr. Math
Math Forum Home || Math Library || Quick Reference || Math Forum Search