Correlation and Regression

Unit: 1
Topic: Correlation and Regression
Subject: Data Analysis and Modeling
BBA 4th Semester
1

Correlation
Introduction:
 Two variables are said to be correlated if the change in
one variable results in a corresponding change in the
other variable.
 The correlation is a statistical tool which studies the
relationship between two variables.
 Correlation analysis involves various methods and
techniques used for studying and measuring the extent
of the relationship between the two variables.
 Correlation is concerned with the measurement of
“strength of association between variables”.
 The degree of association between two or more
variables is termed as correlation.
2

Contd…
 Correlation analysis helps us to decide the strength of the
linear relationship between two variables.
 The word correlation is used to decide the degree of
association between variables.
 If two variables ‘x’ and ‘y’ are so related, the variables in the
magnitude of one variable tend to be accompanied by
variations in the magnitude of the other variable, they are
said to be correlated.
 Thus, correlation is a statistical tool, with the help of which,
we can determine whether or not two or more variables are
correlate and if they are correlated, what is the degree and
direction of correlation.
3

Definition
The correlation is the measure of the extent and
the direction of the relationship between two
variables in a bivariate distribution.
Example:
(i) Height and weight of children.
(ii) An increase in the price of the commodity by a
decrease in the quantity demanded.
Types of Correlation: The following are the types of
correlation
(i) Positive and Negative Correlation
(ii) Simple, Partial and Multiple Correlation
(iii) Linear and Non-linear Correlation
4

Contd…
Correlation first developed by Sir Francis
Galton (1822 – 1911) and then reformulated
by Karl Pearson (1857 – 1936)
Note: The degree of relationship or
association is known as the degree of
relationship.
5

Types of Correlation
i. Positive and Negative correlation: If both the
variables are varying in the same direction i.e. if one
variable is increasing and the other on an average is
also increasing or if as one variable is decreasing, the
other on an average, is also decreasing, correlation is
said to be positive. If on the other hand, the variable
is increasing, the other is decreasing or vice versa,
correlation is said to be negative.
Example 1: a) heights and weights (b) amount of rainfall
and yields of crops (c) price and supply of a
commodity (d) income and expenditure on luxury
goods (e) blood pressure and age
Example 2: a) price and demand of commodity (b) sales
of woolen garments and the days temperature.
6

Contd…
ii. Simple, Partial and Multiple Correlation:
When only two variables are studied, it is a
case of simple correlation. In partial and
multiple correlation, three or more variables
are studied. In multiple correlation three or
more variables are studied simultaneously. In
partial correlation, we have more than two
variables, but consider only two variables to
be influencing each other, the effect of the
other variables being kept constant.
7

Contd…
iii. Linear and Non-linear Correlation: If the
change in one variable tends to bear a
constant ratio to the change in the
other variable, the correlation is said to
be linear. Correlation is said to be non-
linear if the amount of change in one
variable does nor bear a constant ratio
to the amount of change in the other
variable.
8

Methods of Studying Correlation
9
Correlation
Graphical
Method
Scatter
Diagram
Algebraic
Method
Karl Pearson’s
Coefficient of
Correlation

Methods of Studying Correlation
 The following are the methods of determining
correlation
1. Scatter diagram method
2. Karl Pearson’s Coefficient of Correlation
1. Scatter Diagram:
 This is a graphic method of finding out relationship
between the variables.
 Given data are plotted on a graph paper in the form
of dots i.e. for each pair of x and y values we put a
dot and thus obtain as many points as the number of
observations.
 The greater the scatter of points over the graph, the
lesser the relationship between the variables.
10

Scatter Diagram
11
Perfect Positive
Correlation
XO
Y
Perfect Negative
Correlation
XO
Y
Low Degree of
Negative Correlation
XO
Y
Low Degree of
Positive Correlation
XO
Y
High Degree of
Positive Correlation
XO
Y
High Degree of
Negative Correlation
XO
Y
No Correlation
XO
Y
No Correlation
XO
Y

Interpretation
12
 If all the points lie in a straight line, there is either
perfect positive or perfect negative correlation.
 If all the points lie on a straight falling from the lower
left hand corner to the upper right hand corner then the
correlation is perfect positive.
 Perfect positive if r = + 1.
 If all the points lie on a straight falling from the upper
left hand corner to the lower right hand corner then the
correlation is perfect negative.
 Perfect negative if r = -1.
 The nearer the points are to be straight line, the higher
degree of correlation.
 The farthest the points from the straight line, the lower
degree of correlation.
 If the points are widely scattered and no trend is
revealed, the variables may be un-correlated i.e. r = 0.

The Coefficient of Correlation:
 A scatter diagram give an idea about the type of
relationship or association between the variables
under study. It does not tell us about the
quantification of the association between the two.
 In order to quantify the relation ship between the
variables a measure called correlation coefficient
developed by Karl Pearson.
 It is defined as the measure of the degree to which
there is linear association between two intervally
scaled variables.
 Thus, the coefficient of correlation is a number which
indicates to what extent two variables are related , to
what extent variations in one go with the variations
in the other
13

Contd…
14
 The symbol ‘r’ or ‘rₓᵧ’ or ‘rᵧₓ’is denoted in this
method and is calculated by:
 r = {Cov(X,Y) ÷ Sₓ Sᵧ} ………..(i)
 Where Cov(X, Y) is the Sample Covariance
between X and Y. Mathematically it is defined
by
 Cov(X, Y)={∑(X–X̅)(Y–Y̅)} ÷(n – 1)
 Sₓ = Sample standard deviation of X, is given by
 Sₓ = {∑(X – X̅)² ÷ (n – 1)}½
 Sᵧ = Sample standard deviation of Y, is given by
 Sᵧ = {∑(Y – Y̅)² ÷ (n – 1)}½ and X̅ = ∑X ÷ n and Y̅ =
∑Y ÷ n

Interpretation
i. If the covariance is positive, the relationship is
positive.
ii. If the covariance is negative, the relationship is
negative.
iii. If the covariance is zero, the variables are said to be
not correlated.
 Hence the covariance measures the strength of linear
association between considered numerical variables.
 Thus, covariance is an absolute measure of linear
association .
 In order to have relative measure of relationship it is
necessary to compute correlation coefficient .
 Computation of correlation coefficient a relation
developed by Karl Pearson are as follows:
15

Contd…
16
 The formula for sample correlation coefficient (r) is calculated
by the following relation:
If (X – X̅) = x and (Y - Y̅) = y then above formula reduces to:
Example:

Properties of Karl Pearson’s Correlation Coefficient
1. The coefficient of correlation ‘r’ is always a number between -1
and +1 inclusive.
2. If r = +1 or -1, the sample points lie on a straight line.
3. If ‘r’ is near to +1 or -1, there is a strong linear association
4. If ‘r’ is small(close to zero), there is low degree of correlation
5. The coefficient of correlation is the geometric mean of the two
regression coefficients.
Symbolically: r= √(bₓᵧ . bᵧₓ)
Note: It is clear that correlation coefficient is a measure of the
degree to which the association between the two variables
approaches a linear functional relationship.
17

Interpretation of Correlation Coefficient
i. The coefficient of correlation, as obtained by the above formula shall
always lie between +1 to -1.
ii. When r = +1, there is perfect positive correlation between the
variables.
iii. When r = -1, there is perfect negative correlation between the
variables.
iv. When r = 0, there is no correlation.
v. When r = 0.7 to 0.999, there is high degree of correlation.
vi. When r = 0.5 to 0.699, there is a moderate degree of correlation.
vii. When r is less than 0.5, there is a low degree of correlation.
viii. The value of correlation lies in between -1 to +1 i.e.
-1⩽ r ⩽ +1.
ix. The correlation coefficient is independent of the choice of both
origin and scale of observation.
x. The correlation coefficient is a pure number. It is independent of the
units of measurement.
18

Coefficient of Determination
 The coefficient of determination(r²) is the square of the
coefficient of correlation.
 It is the measure of strength of the relationship
between two variables.
 It is subject to more precise interpretation because it
can be presented as a proportion or as a percentage.
 The coefficient of determination gives the ratio of the
explained variance to the total variance.
 Thus, coefficient of determination,
r² = Explained variance ÷ Total variance
Thus, coefficient of determination shows what amount of
variability or change in independent variable is
accounted for by the variability of the dependent
variable.
19

Example
 Example 1: If r = 0.8 then r2 = (0.8)2 = 0.64 or 64%. This means that
based on the sample, 64% of the variation in dependent variable (Y), is
caused by the variations of the independent variable (X). The
remaining 36% variation in Y is unexplained by variation in X. In other
words, the variations other than X could have caused the remaining
36% variations in Y.
 Example 2: While comparing two correlation coefficients, one of which
is 0.4 and the other is 0.8 it is misleading to conclude that the
correlation in the second case is twice as high as correlation in the first
case. The coefficient of determination clearly explains this viewpoint,
since in the case r = 0.4, the coefficient of determination is r² = 0.16
and in the case r = 0.8, the coefficient of determination is r² = 0.64,
from which we conclude that correlation in the second case is four
times as high as the correlation in the first case. If the value of r = 0.8,
we cannot conclude that 80% of the variation in the relative series
(dependent variable). But the coefficient of determination in this case
is r² = 0.64 which implies that only 64% of the variation in the relative
series has been explained by the subject series and the remaining 36%
of the variation is due to other factor.
20

Interpretation
 The closeness of the relationship between two variables as
determined by correlation coefficient r is not proportional.
 The following table gives the values of the coefficient of
determination (r2) for different values of ‘r’.
 It is clear from the above table that as the value of ‘r’
decreases, r2 decreases very rapidly expect in two particular
case r = 0 and r = 1 when we get r2 = r.
 Coefficient of determination (r2) is always non-negative and as
such that it does not tell us about the direction of the
relationship (whether it is positive or negative) between two
series.
21
(r) 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
(r)2 0.01 0.04 0.09 0.16 0.25 0.36 0.49 0.64 0.81 1.00
(r)2in
%
1 % 4 % 9 % 16 % 25% 36 % 49 % 64 % 81 % 100 %

Test of significance of Correlation Coefficient
In order to asses whether the computed
correlation coefficient between considered
variables X and Y are statistically significant or
not t-test can be applied as test statistic. For
this the following steps can be performed:
 Step 1. Formulating the Hypothesis: The
common way of starting a hypothesis is that
“The population correlation coefficient (ρ) is
zero that means there is no correlation
between X and Y variables in the population”.
Null Hypothesis, H₀: ρ = 0 ( No correlation
between X and Y variables in the population)
22

Contd…
Alternative Hypothesis, H₁: ρ ≠ 0 ( There is
correlation between X and Y variables in the
population) (Two tail test)
Or H₁: ρ > 0 (There is positive correlation between
X and Y variables in the population) (One tail test)
Or H₁: ρ < 0 (There is negative correlation between
X and Y variables in the population) (One tail test)
Step 2. Computing the Test Statistic: The test
statistic t for testing the existence of correlation is:
23

Contd…
 where; r = the sample correlation coefficient.
 ρ = the population correlation coefficient which is
hypothesized as zero.
 n= the total number of pair of observations under study.
 Step 3. Decision:
 (i) If the computed value of t is greater than table value of t
at given level of significance (α) with (n – 2) d. f. , we reject
the null hypothesis and conclude that there is evidence of
an association between considered variables.
 (ii) If the computed value of t is less than table value of t at
given level of significance (α) with (n – 2) d. f. , we accept
the null hypothesis and conclude that there is not evidence
of an association between considered variables.
 Example:
24

Simple Linear Regression
 Regression is concerned with the “Prediction” of the most
likely value of one variable when the value of the other
variable is known.
 The term regression literally means “stepping back towards
the average”.
 It was first used by British biometrician Sir Francis Galton
(1822 – 1911).
Definition: Regression analysis is a mathematical measure of
the average relationship between two or more variables in
terms of the original units of the data.
 Thus term regression is used to denote estimation or
prediction of the average value of one variable for a
specified value of the other variable.
 The estimation is done by means of suitable equation,
derived on the basis of available bivariate data. Such an
equation and its geometrical representation is called
regression curve.
25

Contd…
 In regression analysis there are two types of variables
and they are:
 i. Independent and ii. Dependent.
 Dependent variable(Y): The variable whose value is
influenced or is to be predicted is called dependent
variable.
 Independent variable(X): The variable which influences
the values or is used for prediction is called
independent variable.
 In regression analysis independent variable is known as
regressor or predictor or explanatory variable.
 In regression analysis dependent variable is known as
regressed or explained variable.
 Thus the term regression is used to denote estimation
or prediction of the average value of one variable for a
specified value of the other variable.
26

The lines of Regression
 A line fitted to a set of data points to estimates the
relationship between two variables is called regression
line.
 The regression equation of Y on X describes the
changes in the value of Y for given changes in the value
of X.
 The regression equation of X on Y describes the
changes in the value of X for given changes in the value
of Y.
 Hence, an equation for estimating a dependent variable
Y for X from the independent variable X or Y, is called
regression equation of Y on X or X on Y respectively.
 The regression equations of the regression lines, also
called least squares lines are determined by least
square method.
27

Simple Regression Model
• Simple regression line is a straight line that describe
about the dependence of the average value of one
variable on the other.
• Y = β₀ + β₁ X + Ɛ ……….(*)
• Where Y = Dependent or response or outcome variable
(Population)
• X = Independent or explanatory or predictor variable
(Population)
• β₀ = Y- intercept of the model for the population
• β₁ = population slope coefficient or population
regression coefficient. It measures the average rate of
change in dependent variable per unit change in
independent variable.
• Ɛ= Population error in Y for observation.
28

29
One unit change in x
Slope = β₁
β₀
Y- intercept
Error term
Estimated value
of y when x = x₀
x₀ = A specific value of the
independent variable x
Straight line
ŷ = β₀ + β₁ x
An observed value
of y when x = x₀
X
Y

Estimation of Regression Equation
30
Regression Model
Y = β₀ + β₁ X + Ɛ
Regression
Equation
Y = β₀ + β₁ X
Unknown
parameter
β₀ & β₁
Estimated
Regression
equation
ŷ = b₀ + b₁ x
Sample Statistics
b₀ & b₁
Sample Data:
x y
x₁ y₁
x₂ y₂
. .
. .
xn yn
b₀ & b₁
Provides estimates
of
β₀ & β₁

Model
 Linear regression model is
 Y = β₀ + β₁ X + Ɛ
 Linear regression equation is:
 Y = β₀ + β₁ X
 Sample regression model is
 ŷ= b₀ + b₁x + e
 Sample regression equation is
 ŷ= b₀ + b₁x
 Where b₀ = sample y intercept,
 b₁= sample slope coefficient
 x= independent variable
 y= dependent variable
 ŷ= estimated value of dependent variable for a given value
of independent variable.
 e = error term = y - ŷ
31

Least square graphically
32
e1
e3
e2
e5
e4
y2= b0+ b2x2 + e2
y1= b0+ b1x1 +e1
ŷi= b0+ b1xi

Least squares methods
• Let ŷ= b₀ + b₁x …..(1) be estimated linear
regression equation of y on x of the regression
equation Y = β₀ + β₁ X .
• By using the principles of least square, we can
get two normal equations of regression
equation (1) are as:
• ∑y = nb₀ +b₁ ∑x………(2)
• ∑xy = b₀∑x₁ + b₁∑x₂…….(3)
• By solving equations (2) & (3) we get the value
of b₀ & b₁ as:
33

Contd…
• The computational formula for y intercept b₀ as follows:
34
• After finding the value of b₀ & b₁, we get the required fitted
regression model of y on x as ŷ= b₀ + b₁x .

Measures of variation
• There are three measures of variations.
• They are as follows:
i. Total Sum of Squares (SST):It is a measures of
variation in the values of dependent variable (y)
around their mean value (y̅). That is
• SST = ∑(y – y̅)² = ∑y² - (∑y)²/n = ∑y² - n.y̅².
• Note: The total sum of squares or the total variation
is divided into the sum of two components. One is
explained variation due to the relationship between
the considered dependent variable (y) and the
independent variable (x) and the other is unexplained
variation which might be developed due to some
other factors other than the relationship between
variable x and y.
35

Contd…
ii. Regression Sum of Squares( SSR): The
regression sum of squares is the sum of the
squared differences between the predicted
value of y and the mean value of y.
• SSR = ∑(ŷ - y̅)² = b₀.∑y+b₁ ∑xy – (∑y)²/n =
b₀.∑y+b₁∑xy – n.y̅²
ii. Error Sum of Squares (SSE): The error sum of
square is computed as the sum of the
squared differences between the observed
value of y and the predicted value of y i.e.
• SSE = ∑(y – ŷ)² = ∑y²- b₀∑y – b₁∑xy.
36

Contd…
38
Relationship: From the above figures the
relationship of SST, SSR and SSE are as follows
SST = SSR + SSE………………(i)
Where: SST = Total sum of square
SSR = Regression sum of squares
SSE = Error sum of squares
• The fit of the estimated regression line would
be best if every value of the dependent variable
y falls on the regression line.

Contd….
• If SSE = 0 i. e. e = (y – ŷ) = 0 then SST = SSR.
• For the perfect fit of the regression model, the
ratio of SSR to SST must be equal to unity i. e.
If SSE = 0 then the model would be perfect.
• If SSE would be larger, the fit of the regression
line would be poor.
• Note: Largest value of SSE the regression line
would be poor and if SSE = 0 the regression
line would be perfect.
39

Coefficient of Determination (r²)
• The coefficient of determination measures the
strength or extent of the association that exists
between dependent variable (y) and independent
variable (x).
• It measures the proportion of variation in the
dependent variable (y) that is explained by
independent variable of the regression line.
• Coefficient of variation measures the total
variation in the dependent variable due to the
variation in the independent variable and it is
denoted b r².
• r² = SSR/SST but SST = SSE + SSR
• then SSR = SST - SSE
• r² = 1 – (SSE/SST) = (b₀.∑y+b₁∑xy – n.y̅²)/(∑y² – ny̅²).
40

Contd…
• Note:
i. Coefficient of determination is the square of
coefficent of correlation.
then r = ±√r²
ii. If the regression coefficient (b₁) is negative then
take the negative sign
iii. If the regression coefficient (b₁) is positive then
take the positive sign
• Adjusted coefficient of determination: The
adjusted coefficient of determination is
calculated by using the following relation:
•
41

The Standard Error of Estimates
 The standard error of estimate of Y on X, denoted by Sᵧᵪ, measures the
average variation or scatteredness of the observes data point around the
regression line. It is used to measure the reliability of the regression
equation. It is calculated by the following relation:
Interpretation of standard error of the estimate:
i. If the standard error of estimate is larger, there is greater scattering or
dispersion of the data points around the fitted line then the regression
line is poor.
ii. If the standard error is small, then there is less variation of the observed
data around the regression line. So the regression line will be better for
the predicting the dependent variable.
iii. If the standard error is zero, it is expected that the estimating equation
would be perfect estimator of the dependent variable.
42

Test of Significance of Regression Coefficient in
Simple Linear Regression Model
• To test the significance of regression
coefficient of the simple linear regression
model Y = β₀ + β₁ X + Ɛ, the following statistical
test have been applied.
i. t- test for significance in simple linear
regression.
ii. F-test for significance in simple linear
regression.
43

(i) t- test for Significance in Simple Linear Regression
• t-test is applied whether the regression
coefficient β₁ is statistically significant or not.
• The process of setting Hypothesis are as follows:
• Setting of Hypothesis:
• Null Hypothesis, H₀: β₁=0 (The population slope
(β₁) is zero between two variables X and Y in the
population.)
• Alternative Hypothesis, H₁: β₁ ≠ 0 (The population
slope (β₁) is not zero between two variables X and
Y in the population.) or H₁: β₁ > 0 or H₁: β₁ < 0
44

Contd…
• Test statistic: Under H₀ the test statistic is:
45
• Where
• This test statistic t follows t- distribution with (n – 2) d. f.
• Decisions: (i) If tcal < t tab at α % level of significance with (n-2) d.
f. then we accept H₀.
• (ii) If tcal >t tab at α % level of significance with (n-2) d. f. then we
reject H₀ then we accept H₁.

Confidence Interval Estimating for β₁
• Another way for the linear relationship between
the variables X and Y, we can construct
confidence Interval (C. I.) estimate of β₁.
• By the help of C. I. we conclude that whether the
hypothesized value (β₁= 0) is included or not.
• For this the following formula is used:
• C. I. for β₁ = b₁ ± t(n – 2).Sb₁
• Conclusion: If this confidence interval does not
include 0(zero), then we can conclude that there
is significant relationship between the variables X
and Y.
• Example:
46

(ii) F-test for Significant in Simple Linear Regression
• F- test based on F-probability distribution can
also be applied in order to test for the
significance in regression.
• The process of setting Hypothesis are as follows:
• Setting of Hypothesis:
• Null Hypothesis, H₀: β₁=0 (The population slope
(β₁) is zero between two variables X and Y in the
population.)
• Alternative Hypothesis, H₁: β₁ ≠ 0 (The population
slope (β₁) is not zero between two variables X and
Y in the population.) or H₁: β₁ > 0 or H₁: β₁ < 0
47

Contd…
• Test statistic: F is defined as the ratio of
regression mean square (MSR) to the error mean
square (MSE).
• Where, MSR = SSR/k and MSE = SSE/(n-k-1)
• k= No. of independent variables in the regression
model. The value of k = 1 for simple linear
regression model as it has only one predictor
variable x.
• SSR = regression sum of squares = ∑(ŷ – y̅)²
• SSE = error sum of squares = ∑(y – ŷ)²
• The test statistic F- follows F-distribution with (n –
k – 1) i. E. (n – 2) d. f. With k = 1.
48

Contd…
• The ANOVA Table for F- statistic are summarized as:
49
Sources of
variation
Sum of
squares
d. f. Mean square F- ratio
Regression SSR 1 MSR = SSR/1 F = MSR / MSE
Error SSE (n – 2) MSE = SSE/n - 2
Total SST (n – 1)
• Decisions:
i. (i) If Fcal < Ftab at α % level of significance and F with 1
d. f. in the numerator and (n-2) d. f. in the
denominator then we accept H₀.
ii. (ii) If Fcal > Ftab at α % level of significance and F with 1
d. f. in the numerator and (n-2) d. f. in the
denominator then we reject H₀ the accept H₁.

Contd…
iii. Using p- value we reject H₀ if p- value is less
than α.
• Note:
i. F- test will provide same conclusion as provided
by the t-test for only one independent variable.
ii. For simple linear regression; if t-test indicates β₁
≠ 0 and hence the F-test will also show a
significance relationship.
iii. However only the F-test can be used to test for
an overall significant relationship for the
regression with more than one independent
variable.
50

Confidence Interval Estimating of the Mean value of y.
• A point estimate is a single numerical estimate of
y is produces without any indication of ite
accuracy.
• A point estimate provides no sense of how far off
it may be from the population parameter.
• To determine the information a prediction or
confidence interval is developed.
• Prediction interval are used to predict particular y
values for a given value of x.
• Confidence intervals are used to estimate the
mean value of y for a given value of x.
• The point estimate of the mean value of y is same
as the point estimate of an individual value of y.
51

Contd…
• The formula to compute the confidence interval estimate for the mean
value of y is:
52
• The formula to compute the prediction interval estimate of an individual
value of y is:
• Where; ŷ = estimated or predicted value of the dependent variable
for a given value of independent variable.
• Sᵧᵪ = standard error of estimate
• t(n-2)= tabulated value of t for (n- 2) d. f. and α level of significance.
• h= hat matrix element.
• n = number of pairs of observations or sample size.

Interval Estimates for different values of x
53
x̅
X
Y
A Given X
Prediction interval for
a individual Y
Confidence interval
for the mean of Y

Correlation and Regression

More Related Content

What's hot

Viewers also liked

Similar to Correlation and Regression

Recently uploaded

In this document

Correlation and Regression