Matrix algebra PSY544 – Introduction to Factor Analysis Week 2 https://image.freepik.com/free-vector/matrix-silhouette-character-in-black-and-white_72147494638.jp g Prologue •Matrix algebra is a framework for manipulating collections of numbers or algebraic symbols. • •Factor model is an algebraic system. If you understand the way it is communicated, you gain a better appreciation of what is going on. • •We have already seen the common factor model representing the structure of score xij – this model applies to every xij in the data matrix X. Matrix algebra will allow us to express that. Definitions Definitions p = 3 manifest variables N = 4 subjects Definitions •Order: The size of a matrix. •A matrix with N rows and p columns is of order N x p • •Square matrix: A matrix with the same number of rows and columns •Vector: A matrix with a single column (column vector) or a single row (row vector) • Definitions Arithmetic operations Multiplication Multiplication •Matrix multiplication is not what you might think it is! •Matrix multiplication: A and B have to be conformable for multiplication. Matrices are conformable if the number of columns in the first matrix equals the number of rows in the second. • •A B = AB = C • n x p p x m n x m •The product matrix has as many rows as the first matrix and as many columns as the second matrix. •In this case, B was being pre-multiplied by A (and A was being post-multiplied by B) Multiplication Multiplication A B C Multiplication •Matrix multiplication is associative: A(BC) = AB(C) •Matrix multiplication is not commutative: AB ≠ BA •Matrix multiplication is distributive: A(B+C) = AB + AC; (B+C)A = BA + CA •The transpose of a product of matrices equals the product of the transposes in reverse order: (AB)’ = B’ A’ Kinds of matrices Kinds of matrices Kinds of matrices Kinds of matrices Kinds of matrices Kinds of matrices •Orthogonal matrix: A square matrix T is orthogonal if TT’ = I or T’T = I • •Correlation matrix is a square, symmetric matrix with unit diagonals and off-diagonal elements that satisfy -1 ≤ rij ≤ 1. Also, it has to be nonnegative definite (we will define that later) Functions of matrices •The Determinant of a square matrix A is a scalar function of the elements of A. It is denoted as |A| or det(A) and is a single number (scalar). •The determinant has many functions which we will not cover here (neither will we cover the definition or computation) •If a matrix has determinant equal to zero, the matrix is called singular. This is an indication that there is redundancy among the rows / columns of the matrix – if the determinant is zero, some columns (or rows) of the matrix can be expressed as linear combinations of other columns (rows). In other words, the columns (rows) are linearly dependent. • Functions of matrices Functions of matrices •Trace: The trace of a square matrix A, tr(A), is the sum of its diagonal elements. • •Rank: The column rank of A is equal to the total number of linearly independent columns of A. The row rank of A is equal to the total number of linearly independent rows of A. • The rank of an N x K matrix is at most the minimum of N or K, min(N,K) •A matrix whose rank is equal to min(N,K) is full rank •A matrix whose rank is less than min(N,K) is rank deficient Functions of matrices Functions of matrices •Solving equations: • •Consider the equation Ax = b, where A is a N x N non-singular matrix, b is a N x 1 vector and x is a N x 1 vector. We know the elements of A and b and wish to solve for x: •Ax = b •A-1Ax = A-1b •Ix = A-1b •x = A-1b • • Functions of matrices Functions of matrices •Eigenvalues and Eigenvectors •Suppose that S is a square symmetric matrix of order p. If u is a column vector of order p and v is a scalar, such that: •Su = vu •...then v is said to be an eigenvalue (or characteristic root) of S and u is said to be an eigenvector (or characteristic vector) of S. • •S will have p eigenvalues and p associated eigenvectors. Functions of matrices •Eigenvalues and Eigenvectors • •If all p eigenvalues are positive, the matrix is positive definite. If one or more eigenvalues are zero and the rest is positive, the matrix is nonnegative definite. If one or more eigenvalues are negative, the matrix is negative definite. • •The determinant of S, det(S), equals the product of the eigenvalues of S Thus, if one or more eigenvalues are zero, the matrix is singular. • Functions of matrices •Eigenvalues and Eigenvectors • •The eigenvalues can be arranged in descending order as the diagonal elements in a diagonal matrix D, and the corresponding eigenvectors can be arranged as columns of matrix U. Then: U is orthogonal, that is, U’U = I The “eigenstructure” of S can be given in this form: SU = UD It also holds that S = UDU’ • Linear combinations of random variables •Matrix equations are handy for representing linear combinations of random variables •Let x be a column vector of order p containing scores for a random individual on variables x1, x2, …, xp •Let z be a column vector of order m containing scores for a random individual on variables z1, z2, …, zm •We will represent the variables in x as linear functions of the variables in z. Let A be a matrix of order p x m containing coefficients ajk representing the linear effects of zk on xj •Let μ be a column vector of order p containing fixed constants μ1, μ2, …, μp Linear combinations of random variables An intermezzo – expected values •Wiki: “The expected value of random variable is the long-run average value of repetitions of the experiment it represents” • ...so, the expected value is the variable’s mean. • •E[X] = μ An intermezzo – expected values An intermezzo – expected values Covariance matrix •Now suppose that x is a vector of order p containing scores on p variables for a random individual selected from some population, and μ is a vector of order p containing the population means of these p variables. •Then, vector (x – μ) stands for the vector x with the population means subtracted (it represents deviations from the mean) • •Let’s multiply this vector by its transpose: •(x – μ) (x – μ)’ Covariance matrix •Let’s multiply this vector by its transpose: •(x – μ) (x – μ)’ • •…and take the expectation: •E[(x – μ) (x – μ)’] • • Covariance matrix Covariance matrix •E[(x – μ) (x – μ)’] • •The variance-covariance matrix is a p x p symmetric matrix with variances on the diagonal and covariances off the diagonal