Introduction to Optimization methods lecture 1

Chapter 3 Transformations
1. Linear Transformations
2. Eigenvalues and Eigenvectors
3. Orthogonal Projections
4. Quadratic Forms
5. Matrix Norms
Ü©( (>f‰EŒÆ) Chapter 3 Transformations 1 / 16

Linear Transformations
Definition (linear transformations)
A function L : Rn
→ Rm
is called a linear transformation if:

A function L : Rn
→ Rm
1 L(ax) = aL(x) for every x ∈ Rn
and a ∈ R.

A function L : Rn
→ Rm
and a ∈ R.
2 L(x + y) = L(x) + L(y) for every x, y ∈ Rn
.

A function L : Rn
→ Rm
and a ∈ R.
.
Definition (matrix representation)
Suppose that x ∈ Rn
, and x0
is the representation of x with respect to the given
basis for Rn
. If y = L(x), and y0
is the representation of y with respect to the
given basis for Rm
, then y0
= Ax0
. A is called the matrix representation of L
with respect to the given bases for Rn
and Rm
.

A function L : Rn
→ Rm
and a ∈ R.
.
Definition (matrix representation)
Suppose that x ∈ Rn
, and x0
is the representation of x with respect to the given
basis for Rn
. If y = L(x), and y0
is the representation of y with respect to the
given basis for Rm
, then y0
= Ax0
. A is called the matrix representation of L
with respect to the given bases for Rn
and Rm
.
F Particularly, for the natural bases for Rn
and Rm
, the matrix representation
A satisfies L(x) = Ax.

Definition (transformation matrix)
Let {e1, e2, . . . , en} and {e0
1, e0
2, . . . , e0
n} be two bases for Rn
. Define the matrix
T = [e0
1, e0
2, . . . , e0
n]−1
[e1, e2, . . . , en], or equivalently
[e1, e2, . . . , en] = [e0
1, e0
2, . . . , e0
n]T ,
T is called the transformation matrix.

Let {e1, e2, . . . , en} and {e0
1, e0
2, . . . , e0
. Define the matrix
T = [e0
1, e0
2, . . . , e0
n]−1
[e1, e2, . . . , en] = [e0
1, e0
2, . . . , e0
n]T ,
Example
For any u ∈ Rn
, let x (resp. x0
) be the coordinates of u with respect to
{e1, e2, . . . , en} (resp. {e0
1, e0
2, . . . , e0
n}). Then, x0
= T x.

Let {e1, e2, . . . , en} and {e0
1, e0
2, . . . , e0
. Define the matrix
T = [e0
1, e0
2, . . . , e0
n]−1
[e1, e2, . . . , en] = [e0
1, e0
2, . . . , e0
n]T ,
Example
For any u ∈ Rn
, let x (resp. x0
{e1, e2, . . . , en} (resp. {e0
1, e0
2, . . . , e0
n}). Then, x0
= T x.
Similarity: A linear transformation L : Rn
→ Rm
.

Let {e1, e2, . . . , en} and {e0
1, e0
2, . . . , e0
. Define the matrix
T = [e0
1, e0
2, . . . , e0
n]−1
[e1, e2, . . . , en] = [e0
1, e0
2, . . . , e0
n]T ,
Example
For any u ∈ Rn
, let x (resp. x0
{e1, e2, . . . , en} (resp. {e0
1, e0
2, . . . , e0
n}). Then, x0
= T x.
→ Rm
.
Let A (resp. B) be its representation of {e1, e2, . . . , en} (resp. {e0
1, e0
2, . . . , e0
n}).

Let {e1, e2, . . . , en} and {e0
1, e0
2, . . . , e0
. Define the matrix
T = [e0
1, e0
2, . . . , e0
n]−1
[e1, e2, . . . , en] = [e0
1, e0
2, . . . , e0
n]T ,
Example
For any u ∈ Rn
, let x (resp. x0
{e1, e2, . . . , en} (resp. {e0
1, e0
2, . . . , e0
n}). Then, x0
= T x.
→ Rm
.
1, e0
2, . . . , e0
n}).
Let y = Ax and y0
= Bx0
.

Let {e1, e2, . . . , en} and {e0
1, e0
2, . . . , e0
. Define the matrix
T = [e0
1, e0
2, . . . , e0
n]−1
[e1, e2, . . . , en] = [e0
1, e0
2, . . . , e0
n]T ,
Example
For any u ∈ Rn
, let x (resp. x0
{e1, e2, . . . , en} (resp. {e0
1, e0
2, . . . , e0
n}). Then, x0
= T x.
→ Rm
.
1, e0
2, . . . , e0
n}).
Let y = Ax and y0
= Bx0
.
∴ y0
= T y = T Ax = Bx0
= BT x,

Let {e1, e2, . . . , en} and {e0
1, e0
2, . . . , e0
. Define the matrix
T = [e0
1, e0
2, . . . , e0
n]−1
[e1, e2, . . . , en] = [e0
1, e0
2, . . . , e0
n]T ,
Example
For any u ∈ Rn
, let x (resp. x0
{e1, e2, . . . , en} (resp. {e0
1, e0
2, . . . , e0
n}). Then, x0
= T x.
→ Rm
.
1, e0
2, . . . , e0
n}).
Let y = Ax and y0
= Bx0
.
∴ y0
= T y = T Ax = Bx0
= BT x,
hence, T A = BT , or A = T −1
BT .

Eigenvalue and Eigenvector
Definition (eigenvalue and eigenvector)
Let A ∈ Rn×n
. A scalar λ ∈ C and a vector v 6= 0 satisfying Av = λv are said to
be an eigenvalue and an eigenvector of A.

Let A ∈ Rn×n
F Calculation of eigenvalues/spectrum of A ⇐⇒
det[λI − A] = λn
+ an−1λn−1
+ · · · + a1λ + a0 = 0. (characteristic equation)

Let A ∈ Rn×n
+ an−1λn−1
Corollary
If det[λI − A] = 0 has n distinct roots {λi}n
i=1. Then, there exist n linearly
independent vectors {vi}n
i=1 such that Avi = λivi, i = 1, . . . , n.

Let A ∈ Rn×n
+ an−1λn−1
Corollary
Theorem (let A ∈ Rn×n
be a square matrix)

Let A ∈ Rn×n
+ an−1λn−1
Corollary
be a square matrix)
A is similar to diagonal matrix ⇐⇒ A has n linearly independent
eigenvectors {vi}n
i=1.

Let A ∈ Rn×n
+ an−1λn−1
Corollary
be a square matrix)
A is similar to diagonal matrix ⇐⇒ A has n linearly independent
eigenvectors {vi}n
i=1.
A is similar to diagonal matrix ⇐= A has n distinct eigenvalues {λi}n
i=1.

procedures for diagonalizing a matrix

1 calculate the eigenvalues of A, i.e., {λi}n
i=1;

i=1;
2 calculate the eigenvectors of A, i.e., {vi}n
i=1;

i=1;
i=1;
3 let T = [v1, v2, . . . , vn], Λ = diag(λ1, λ2, . . . , λn), then A = T ΛT −1
.

i=1;
i=1;
.
Theorem (symmetric matrix: A ∈ Rn×n
satisfying A = A>
)

i=1;
i=1;
.
satisfying A = A>
)
All eigenvalues of a real symmetric matrix A ∈ Rn×n
are real.

i=1;
i=1;
.
satisfying A = A>
)
are real.
Any real symmetric matrix A ∈ Rn×n
has a n mutually orthogonal
eigenvectors. (proof on blackboard)

i=1;
i=1;
.
satisfying A = A>
)
are real.
has a n mutually orthogonal
eigenvectors. (proof on blackboard)
Definition (orthogonal matrix)
A matrix whose transpose is its inverse is said to be an orthogonal matrix, i.e.,
T −1
= T >
.

Theorem (diagonalize symmetric matrix)
has a diagonal form A = T ΛT >
with T as
orthogonal matrix and Λ as diagonal matrix.

with T as
Procedures for diagonalizing a real symmetry matrix

with T as
i=1;

with T as
i=1;
i=1;

with T as
i=1;
i=1;
3 normalize individually the eigenvectors of λi, T = [u1, u2, . . . , un]

with T as
i=1;
i=1;
3 normalize individually the eigenvectors of λi, T = [u1, u2, . . . , un]
4 let T = [u1, . . . , un], Λ = diag(λ1, . . . , λn), then A = T ΛT −1
= T ΛT >
.

Orthogonal Projections
Definition (subspace)
A set V ⊆ Rn
a subspace if x1, x2 ∈ V =⇒ αx1 + βx2 ∈ V, ∀α, β ∈ R. The
dimension of V, denoted by dimV, is the maximum number of linearly
independent vectors in V.

A set V ⊆ Rn
Definition (orthogonal complement)
If V ⊆ Rn
a subspace, then the orthogonal complement of V, denoted by V⊥
,
consists of all vectors that are orthogonal to every vector in V, i.e.,
V⊥
= {x | v>
x = 0, ∀v ∈ V}.

A set V ⊆ Rn
If V ⊆ Rn
,
V⊥
= {x | v>
x = 0, ∀v ∈ V}.
F V ⊥
is also a subspace of Rn
. V and V⊥
span Rn
(or Rn
is the direct sum of V
and V⊥
), i.e., Rn
= V ⊕ V⊥
. Concisely, every x ∈ Rn
can be represented
uniquely as
[orthogonal decomposition] x = x1 + x2, where x1 ∈ V, x2 ∈ V⊥
.

A set V ⊆ Rn
If V ⊆ Rn
,
V⊥
= {x | v>
x = 0, ∀v ∈ V}.
F V ⊥
is also a subspace of Rn
. V and V⊥
span Rn
(or Rn
is the direct sum of V
and V⊥
), i.e., Rn
= V ⊕ V⊥
. Concisely, every x ∈ Rn
can be represented
uniquely as
[orthogonal decomposition] x = x1 + x2, where x1 ∈ V, x2 ∈ V⊥
.
F x1 (resp. x2) is orthogonal projections of x onto V (resp. V⊥
).

Definition (orthogonal projections)
A linear transformation P is an orthogonal projector onto V if for all x ∈ Rn
, we
have P x ∈ V and x − P x ∈ V⊥
.

, we
.
Definition (range and null space of matrix A ∈ Rm×n
)

, we
.
)
The range (or image) of A: R(A) = {Ax | x ∈ Rn
};

, we
.
)
};
The nullspace (or kernel) of A: N(A) = {x ∈ Rn
| Ax = 0}.

, we
.
)
};
| Ax = 0}.
F Both R(A) and N(A) are subspaces.

, we
.
)
};
| Ax = 0}.
F Both R(A) and N(A) are subspaces.
Lemma
For matrix A ∈ Rm×n
, R(A)⊥
= N(A>
) and N(A)⊥
= R(A>
). (proof on
blackboard)

Theorem (property of projection)
A matrix P ∈ Rn×n
is an orthogonal projector onto the subspace V = R(P )
⇐⇒ P 2
= P = P >
. (proof on blackboard)

⇐⇒ P 2
= P = P >
Definition (quadratic form)
f : Rn
→ R is a quadratic form ⇐⇒ f(x) = x>
Qx with Q ∈ Rn×n
.

⇐⇒ P 2
= P = P >
Definition (quadratic form)
f : Rn
→ R is a quadratic form ⇐⇒ f(x) = x>
Qx with Q ∈ Rn×n
.
F w.l.o.g, Q is assumed to be symmetric. If Q is asymmetric, how?

Quadratic Form
A quadratic form x>
Qx is said to be
positive definite ⇐⇒ x>
Qx > 0, ∀x 6= 0 ⇐⇒ Q 0;
positive semidefinite ⇐⇒ x
Qx ≥ 0, ∀x 6= 0 ⇐⇒ Q 0;
negative definite ⇐⇒ x
Qx 0, ∀x 6= 0 ⇐⇒ Q ≺ 0;
negative semidefinite ⇐⇒ x
Qx ≤ 0, ∀x 6= 0 ⇐⇒ Q 0.
Ü©( (f‰EŒÆ) Chapter 3 Transformations 10 / 16

Quadratic Form
A quadratic form x
Qx is said to be
positive definite ⇐⇒ x
Qx 0, ∀x 6= 0 ⇐⇒ Q 0;
Qx ≥ 0, ∀x 6= 0 ⇐⇒ Q 0;
Qx 0, ∀x 6= 0 ⇐⇒ Q ≺ 0;
Qx ≤ 0, ∀x 6= 0 ⇐⇒ Q 0.
Let Q ∈ Rn×n
, and Qp =





qi1j1
qi1j2
· · · qi1jp
qi2j1
qi2j2
· · · qi2jp
.
.
.
.
.
.
...
.
.
.
qipj1
qipj2
· · · qipjp





,
1 ≤ i1 ≤ · · · ≤ ip ≤ n,
1 ≤ j1 ≤ · · · ≤ jp ≤ n.

Quadratic Form
A quadratic form x
Qx is said to be
Qx 0, ∀x 6= 0 ⇐⇒ Q 0;
Qx ≥ 0, ∀x 6= 0 ⇐⇒ Q 0;
Qx 0, ∀x 6= 0 ⇐⇒ Q ≺ 0;
Qx ≤ 0, ∀x 6= 0 ⇐⇒ Q 0.
Let Q ∈ Rn×n
, and Qp =





qi1j1
qi1j2
· · · qi1jp
qi2j1
qi2j2
· · · qi2jp
.
.
.
.
.
.
...
.
.
.
qipj1
qipj2
· · · qipjp





,
1 ≤ i1 ≤ · · · ≤ ip ≤ n,
1 ≤ j1 ≤ · · · ≤ jp ≤ n.
Definition (minor, principal minor, leading principal minor)

Quadratic Form
A quadratic form x
Qx is said to be
Qx 0, ∀x 6= 0 ⇐⇒ Q 0;
Qx ≥ 0, ∀x 6= 0 ⇐⇒ Q 0;
Qx 0, ∀x 6= 0 ⇐⇒ Q ≺ 0;
Qx ≤ 0, ∀x 6= 0 ⇐⇒ Q 0.
Let Q ∈ Rn×n
, and Qp =





qi1j1
qi1j2
· · · qi1jp
qi2j1
qi2j2
· · · qi2jp
.
.
.
.
.
.
...
.
.
.
qipj1
qipj2
· · · qipjp





,
1 ≤ i1 ≤ · · · ≤ ip ≤ n,
1 ≤ j1 ≤ · · · ≤ jp ≤ n.
p-order minor of Q: detQp.

Quadratic Form
A quadratic form x
Qx is said to be
Qx 0, ∀x 6= 0 ⇐⇒ Q 0;
Qx ≥ 0, ∀x 6= 0 ⇐⇒ Q 0;
Qx 0, ∀x 6= 0 ⇐⇒ Q ≺ 0;
Qx ≤ 0, ∀x 6= 0 ⇐⇒ Q 0.
Let Q ∈ Rn×n
, and Qp =





qi1j1
qi1j2
· · · qi1jp
qi2j1
qi2j2
· · · qi2jp
.
.
.
.
.
.
...
.
.
.
qipj1
qipj2
· · · qipjp





,
1 ≤ i1 ≤ · · · ≤ ip ≤ n,
1 ≤ j1 ≤ · · · ≤ jp ≤ n.
p-order principal minor of Q: detQp with ik = jk for all k = 1, . . . , p.

Quadratic Form
A quadratic form x
Qx is said to be
Qx 0, ∀x 6= 0 ⇐⇒ Q 0;
Qx ≥ 0, ∀x 6= 0 ⇐⇒ Q 0;
Qx 0, ∀x 6= 0 ⇐⇒ Q ≺ 0;
Qx ≤ 0, ∀x 6= 0 ⇐⇒ Q 0.
Let Q ∈ Rn×n
, and Qp =





qi1j1
qi1j2
· · · qi1jp
qi2j1
qi2j2
· · · qi2jp
.
.
.
.
.
.
...
.
.
.
qipj1
qipj2
· · · qipjp





,
1 ≤ i1 ≤ · · · ≤ ip ≤ n,
1 ≤ j1 ≤ · · · ≤ jp ≤ n.
p-order principal minor of Q: detQp with ik = jk for all k = 1, . . . , p.
p-order leading principal minors of Q: detQp with ik = jk = k for
k = 1, . . . , p.

Quadratic Form
How to check a symmetric matrix Q is positive definite?

Quadratic Form
1 Q 0 ⇐⇒ all eigenvalues of Q are positive (i.e., {λi 0}n
i=1).

Quadratic Form
i=1).
2 Q 0 ⇐⇒ all leading principal minors of Q are positive. (Sylvester rule)

Quadratic Form
i=1).
hint of proof: by denoting ∆i the i-order leading principal minors of Q, there
exists invertible matrix V such that
x
Qx
x=V x̃
=
=
=
=
=
=
∆0
∆1
x̃2
1 +
∆1
∆2
x̃2
2 + · · · +
∆n−1
∆n
x̃2
n.

Quadratic Form
i=1).
x
Qx
x=V x̃
=
=
=
=
=
=
∆0
∆1
x̃2
1 +
∆1
∆2
x̃2
2 + · · · +
∆n−1
∆n
x̃2
n.
F if Q is asymmetric, Sylvester rule cannot be used.

Quadratic Form
i=1).
x
Qx
x=V x̃
=
=
=
=
=
=
∆0
∆1
x̃2
1 +
∆1
∆2
x̃2
2 + · · · +
∆n−1
∆n
x̃2
n.
Example (counterexample)

Quadratic Form
i=1).
x
Qx
x=V x̃
=
=
=
=
=
=
∆0
∆1
x̃2
1 +
∆1
∆2
x̃2
2 + · · · +
∆n−1
∆n
x̃2
n.
Q =

1 0
−4 1

,

Quadratic Form
i=1).
x
Qx
x=V x̃
=
=
=
=
=
=
∆0
∆1
x̃2
1 +
∆1
∆2
x̃2
2 + · · · +
∆n−1
∆n
x̃2
n.
Q =

1 0
−4 1

,
Although ∆1 = 1 0 and ∆2 = detQ = 1 0, Q 0
(∵ x = [1, 1]
=⇒ x
Qx = −2 0).

Inner Products and Norms
Theorem
Q 0 =⇒ all leading principal minors of Q are nonnegative.

Theorem
F The above Theorem is not a sufficient condition.

Theorem

Theorem
Q =


2 2 2
2 2 2
2 2 0

.

Theorem
Q =


2 2 2
2 2 2
2 2 0

.
Although ∆1 = 2, ∆2 = 0 ∆3 = 0, Q 0.
(∵ x = [1, 1, −2]
⇔ x
Qx 0).

Theorem
Q =


2 2 2
2 2 2
2 2 0

.
Although ∆1 = 2, ∆2 = 0 ∆3 = 0, Q 0.
(∵ x = [1, 1, −2]
⇔ x
Qx 0).
Theorem (positive semidefinite)

Theorem
Q =


2 2 2
2 2 2
2 2 0

.
Although ∆1 = 2, ∆2 = 0 ∆3 = 0, Q 0.
(∵ x = [1, 1, −2]
⇔ x
Qx 0).
Q 0 ⇐⇒ all principal minors of Q are nonnegative.

Theorem
Q =


2 2 2
2 2 2
2 2 0

.
Although ∆1 = 2, ∆2 = 0 ∆3 = 0, Q 0.
(∵ x = [1, 1, −2]
⇔ x
Qx 0).
Q 0 ⇐⇒ all eigenvalues of Q are nonnegative.

Theorem
Q =


2 2 2
2 2 2
2 2 0

.
Although ∆1 = 2, ∆2 = 0 ∆3 = 0, Q 0.
(∵ x = [1, 1, −2]
⇔ x
Qx 0).
Q 0 ⇐⇒ all eigenvalues of Q are nonnegative.
proof in linear algebra textbook.

Matrix Norms
Definition (matrix norm)
The norm of a matrix A is a function satisfying the following conditions:

Matrix Norms
1 kAk 0 if A 6= 0, and k0k = 0, where 0 is a zero matrix.

Matrix Norms
2 kcAk = |c|kAk, ∀c ∈ R.

Matrix Norms
3 kA + Bk ≤ kAk + kBk and kABk ≤ kAkkBk.

Matrix Norms
F e.g., Frobenius norm (i.e., F-norm): kAkF =
m
P
i=1
n
P
j=1
a2
ij
!1/2
.

Matrix Norms
F e.g., Frobenius norm (i.e., F-norm): kAkF =
m
P
i=1
n
P
j=1
a2
ij
!1/2
.
induced matrix norms
Let k · k(n) and k · k(m) be vector norms on Rn
and Rm
. The induced matrix
norm satisfies: kAxk(m) ≤ kAkkxk(n) for all A ∈ Rm×n
, x ∈ Rn
, i.e.,
kAk = max
x6=0
kAxk(m)
kxk(n)
= max
kxk(n)=1
kAxk(m).

Matrix Norms
Definition (Rayleigh’s inequality)
Let λmax(Q) and λmin(Q) be the maximal and minimal eigenvalues of symmetric
matrix Q ∈ Rn×n
. Then, λmin(Q)kxk2
2 ≤ kxk2
Q ≤ λmax(Q)kxk2
2 for all x 6= 0.
(proof: quadratic form in linear algebra.)

Matrix Norms
matrix Q ∈ Rn×n
2 ≤ kxk2
Q ≤ λmax(Q)kxk2
2 for all x 6= 0.
Example (spectral norm: i.e., by setting k · k(n) and k · k(m) as 2-norm)
kAk =
√
λ1, where λ1 is the largest eigenvalue of A
A.

Matrix Norms
matrix Q ∈ Rn×n
2 ≤ kxk2
Q ≤ λmax(Q)kxk2
2 for all x 6= 0.
kAk =
√
A.
proof. ∵ A
A is symmetric and positive semidefinite.

Matrix Norms
matrix Q ∈ Rn×n
2 ≤ kxk2
Q ≤ λmax(Q)kxk2
2 for all x 6= 0.
kAk =
√
A.
proof. ∵ A
Let λ1 ≥ λ2 ≥ · · · ≥ λn ≥ 0 be its eigenvalues,
x1, x2, . . . , xn be the corresponding orthonormal eigenvectors.

Matrix Norms
matrix Q ∈ Rn×n
2 ≤ kxk2
Q ≤ λmax(Q)kxk2
2 for all x 6= 0.
kAk =
√
A.
proof. ∵ A
For any x ∈ Rn
with kxk2 = 1, it follows by Rayleight’s inequality that

Matrix Norms
matrix Q ∈ Rn×n
2 ≤ kxk2
Q ≤ λmax(Q)kxk2
2 for all x 6= 0.
kAk =
√
A.
proof. ∵ A
For any x ∈ Rn
kAxk2
= hx, A
Axi ≤ λ1kxk2
2 = λ1.

Matrix Norms
matrix Q ∈ Rn×n
2 ≤ kxk2
Q ≤ λmax(Q)kxk2
2 for all x 6= 0.
kAk =
√
A.
proof. ∵ A
For any x ∈ Rn
kAxk2
= hx, A
Axi ≤ λ1kxk2
2 = λ1.
For a unit eigenvector x of A
A corresponding to the eigenvalue λ1,

Matrix Norms
matrix Q ∈ Rn×n
2 ≤ kxk2
Q ≤ λmax(Q)kxk2
2 for all x 6= 0.
kAk =
√
A.
proof. ∵ A
For any x ∈ Rn
kAxk2
= hx, A
Axi ≤ λ1kxk2
2 = λ1.
For a unit eigenvector x of A
A corresponding to the eigenvalue λ1,
kAxk2
= hx, A
Axi = λ1hx, xi = λ1.

Matrix Norm
Example (compute the spectral norm of A)

Matrix Norm
A =

2 1
1 2

, kAk =?

Matrix Norm
A =

2 1
1 2

, kAk =?
hint: |λI2 − A
A| = λ2
− 10λ + 9 = (λ − 1)(λ − 9) = 0,
∴ λ1 = 9, ∴ kAk =
√
λ1 = 3.

Matrix Norm
A =

2 1
1 2

, kAk =?
hint: |λI2 − A
A| = λ2
− 10λ + 9 = (λ − 1)(λ − 9) = 0,
∴ λ1 = 9, ∴ kAk =
√
λ1 = 3.
F spectral radius and spectral norm.

Matrix Norm
A =

2 1
1 2

, kAk =?
hint: |λI2 − A
A| = λ2
− 10λ + 9 = (λ − 1)(λ − 9) = 0,
∴ λ1 = 9, ∴ kAk =
√
λ1 = 3.
spectral radius: ρ(A) = max
1≤i≤n
|λi(A)|. square matrix

Matrix Norm
A =

2 1
1 2

, kAk =?
hint: |λI2 − A
A| = λ2
− 10λ + 9 = (λ − 1)(λ − 9) = 0,
∴ λ1 = 9, ∴ kAk =
√
λ1 = 3.
1≤i≤n
spectral norm: kAk = max
1≤i≤n
p
λi(AA). all matrices.

Matrix Norm
A =

2 1
1 2

, kAk =?
hint: |λI2 − A
A| = λ2
− 10λ + 9 = (λ − 1)(λ − 9) = 0,
∴ λ1 = 9, ∴ kAk =
√
λ1 = 3.
1≤i≤n
spectral norm: kAk = max
1≤i≤n
p
λi(AA). all matrices.
F Typically, ρ(A) ≤ kAk. “=” holds if A is symmetric. e.g.,
A =

0 1
0 0

=⇒ λ1(A) = 0 and λ1(A
A) = 1.

Matrix Norms
Other induced matrix norms:
kAk1 = max
1≤j≤n
m
X
i=1
|aij|, kAk∞ = max
1≤i≤m
n
X
j=1
|aij|.

Matrix Norms
kAk1 = max
1≤j≤n
m
X
i=1
|aij|, kAk∞ = max
1≤i≤m
n
X
j=1
|aij|.
Example (some operator norms k · ka,b for matrix A ∈ Rm×n
)
a = 1 a = 2 a = ∞
b = 1 max
j=1,...,n
kA:,jk1 max
j=1,...,n
kA:,jk2 max
j=1,...,n
kA:,jk∞
b = 2 NP-hard λmax(A
A)
1/2
max
i=1,...,m
kAi,:k2
b = ∞ NP-hard NP-hard max
i=1,...,m
kAi,:k1

Matrix Norms
kAk1 = max
1≤j≤n
m
X
i=1
|aij|, kAk∞ = max
1≤i≤m
n
X
j=1
|aij|.
)
a = 1 a = 2 a = ∞
b = 1 max
j=1,...,n
kA:,jk1 max
j=1,...,n
kA:,jk2 max
j=1,...,n
kA:,jk∞
A)
1/2
max
i=1,...,m
kAi,:k2
i=1,...,m
kAi,:k1
F Other norms (but not the induced matrix norm) for an A ∈ Rn×n
:
kAk∗ =
n
X
i=1
σi, kAkF = max
1≤i≤n
σi, where A = UΣV
is SVD.

Matrix Norms
kAk1 = max
1≤j≤n
m
X
i=1
|aij|, kAk∞ = max
1≤i≤m
n
X
j=1
|aij|.
)
a = 1 a = 2 a = ∞
b = 1 max
j=1,...,n
kA:,jk1 max
j=1,...,n
kA:,jk2 max
j=1,...,n
kA:,jk∞
A)
1/2
max
i=1,...,m
kAi,:k2
i=1,...,m
kAi,:k1
F Other norms (but not the induced matrix norm) for an A ∈ Rn×n
:
kAk∗ =
n
X
i=1
σi, kAkF = max
1≤i≤n
σi, where A = UΣV
is SVD.
Homework (Exercise in text book): 3.17, 3.21, 3.22

Introduction to Optimization methods lecture 1

More Related Content

Similar to Introduction to Optimization methods lecture 1

Recently uploaded

Introduction to Optimization methods lecture 1