‘1Chapter

3Further

development

and

analysis

of

theclassical

linear

regression

model西北大學(xué)《金融計(jì)量學(xué)》Multiple

Linear

Regression‘Before,

we

have

used

the

modelyt

a

bxt

utt=

1,2,...,T

But what

if

our

dependent

(y)

variable depends

on

more than

oneindependent

variable?For

example

the

number

of

cars

sold

might

plausibly

depend

onthe

price

of

carsthe

price

of

public

transportthe

price

of

gasthe

extent

of

the

public’s

concern

about

global

warmingMultiple

Linear

Regression‘Similarly,

stock

returns

might

depend

on

several

factors.

Having

just

one

independent

variable

is

no

good

in

this

case

-

we

want

tohave

more

than

one

x

variable.

It

is

very

easy

to

generalise

the

simplemodel

to

one

with

k-1

regressors

(independent

variables).Multiple

Regression

and

the

Constant

Term?yt

b1b

2

x2t

b3

x3t...b,ktx=1kt,2,..u.t,T1b1

is

the

coefficient

attached

to

the

constant

term.1

Where

is

x1?

It

is

the

constant

term.

In

fact

the

constant

term

is

usuallyrepresented

by

a

column

of

ones

of

length

T:11xb

2

measures

the

effect

of

x

2

on

y

afterNow

we

write

eliminating

the

effects

of

x3

,

x

4,

xk‘Different

Ways

of

Expressing

the

Multiple

Linear

Regression

ModelWe

could

write

out

a

separate

equation

for

every

value

of

t:2

x212

x22b

3

x31b

3

x32......b

k

xk1b

k

xk

2u1u2b

k

xkTuTy1b1by2b1byTb1b2

x2Tb

3

x3T...Different

Ways

of

Expressing

the

Multiple

Linear

Regression

Model‘wherey

is

T

1X

is

T

kb

is

k

1u

is

T

1We

can

write

this

in

matrix

formy

=

Xb

+uInside

the

Matrices

of

theMultiple

Linear

Regression

Model‘e.g.

if

k

is

2,

we

have

2

regressors,

one

of

which

is

a

column

of

ones:T

1T

1yTu1u2222y1yx21x11b1b

2x2T

uT1T

2

2

1How

Do

We

Calculate

the

Parameters

(the

b

Vector)in

this

Case?‘u?2u?Previously,

we

took

the

residual

sum

of

squares,

and

minimised

itw.r.t.

a

and

b.In

the

matrix

notation,

we

haveu?12222u?t1u?

2

u?

.T..u??TuTu2?u?TThe

RSS

would

be

given

byu?1L

u

"u?

?u

u1

?

2

?u?The

OLS

Estimator

fortheMultiple

Regression

Model‘"?

?L

u

u

(

y

xb?)"

(

yxb?)(y"

b?"

x"

)(

y

xb?)y"

xb?

b?"

x"

xb?b?"

x"

xb?L0b?(x"

x)1

x"

yb?y"

y

b?"

x"

yy"

y

b1

2?b?"

x"

yb?2b?kerrors,

s

2,

we

usedT

2s

2u?t

.Standard

Errors

of

the

Coefficient

Estimates

for

theMultiple

Regression

Modela

bxtIn

bivariate

regression

y2

t

ut

,

to

estimate

the

varianceIn

multiple

linear

regression,

wse2use

u"

uT

kIntercept

+

one

coefficient‘#

of

regressors+interceptIt

can

be

proved

that

the

OLS

estimator

of

the

variance

of

bdiagonal

elements

of

s2(

X

"

X

)‘1

,

so

that

the

variance

of

bis

given

by

t1is

the

firstelement,

the

variance

of

b

2

is

the

second

element,

and

…,

and

the

variancof

b

k

is

the

kth

diagonal

element.Standard

Errors

of

the

Coefficient

Estimates

for

theMultiple

Regression

ModelCalculating

Parameter

and

Standard

Error

Estimatesfor

Multiple

Regression

Models:

An

ExampleExample:

The

following

model

with

k=3

is

estimated

over

15

observations:y

b

1

b

2

x2

b

3

x3

uCalculate

the

coefficient

estimates

and

their

standard

errors.s20.91RSS

10.96T

k

15

3(

X

"

X

)and

the

following

data

have

been

calculated

from

the

original

X’s.3.02.0

31..50

6.5

,(

X

"

y1).0

2.2 ,

u

"

u

10.961

16.3.0.55

4.3

0.6-1.0

6.5

4.3

0.6To

calculate

the

standard

errors,

we

n1e9.e8d8an

estimate

of

s

2.1.1b1??2bb?3("x

x)12.0

3.5

-1.0

3"x

y

3.5

1.0

6.5b?

2.24.4Calculating

Parameter

and

Standard

Error

Estimatesfor

Multiple

Regression

Models:

An

Example

(cont’d)‘bis

given

The

variance-covariance

matrix

ofbySE

(b11SE

(b22Var(bVar(bVar(b)

1.83)

0.91)

3.93SE

(b)

1.35)

0.96)

1.9833We

write:19.88

x3ty?

1.101.354.40

x2

t0.961.981.830.91

5.94The

variances

are

on

the

leading

diagonal:Var(b?)1s2

(

X

"

X

)

0.91(

X

"

X

)10.913.20

0.915.39.4203.93Testing

Multiple

Hypotheses:

The

F-test‘

T-test

to

test

single

hypotheses,

i.e.

hypotheses

involving

only

onecoefficient.F-test

is

for

multiple

hypothesesTesting

Multiple

Hypotheses:

The

F-test‘F-test

involves

estimating

2

regressions:The

unrestricted

regression

is

the

one

in

which

the

coefficients

are

freeldetermined

by

the

data,

as

we

have

done

before.The

restricted

regression

is

the

one

in

which

the

coefficients

are

restrici.e.

the

restrictions

are

imposed

on

some

bs.The

F-test:Restricted

and

Unrestricted

Regressions‘ExampleThe

general

regression

isyt

=

b1

+

b2x2t

+

b3x3t

+

b4x4t

+

ut(1)

(unrestricted

regression)some

theory

predicts

that:

b3

+b4

=

1s.t.

b3

+b4

=

1yt

=

b1

+

b2x2t

+

b3x3t

+

b4x4t

+

ut(restricted

regression)The

F-test:Restricted

and

Unrestricted

Regressions‘

We

substitute

the

restriction

(b3

+b4

=

1)

into

the

regression

so

that

it

isautomatically

imposed

on

the

data.b3

+b4

=

1

b4

=

1-

b3

(substitute

into

equation

(1))The

F-test:

Forming

the

Restricted

Regressionyt

=

b1

+

b2x2t

+

b3x3t

+

(1-b3)x4t

+

ut

yt=

b1

+

b2x2t

+

b3x3t

+

x4t

-

b3x4t

+

utGather

terms

in

b’s

together

and

rearrange(yt

-

x4

t

)=

b1

+

b2x2t

+

b3(x3t

-

x4t)

+

ut

This

is

the

restricted

regression.

We

actually

estimate

it

by

creating

two

nvariables,

call

them,

say,

Pt

and

Qt.Pt

=

yt

-

x4tQt

=

x3t

-

x4tsoPt

=

b1

+

b2x2t

+

b3

Qt

+

ut

is

the

restricted

regression

we

actually

estimate.‘Calculating

the

F-Test

Statistic‘The

test

statistic

is

given

byF

test

statisticRRSS

URSS

T

k

URSS

mwhere

URSS

=

RSS

from

unrestricted

regressionRRSS

=

RSS

from

restricted

regressionm

=

number

of

restrictionsT

=

number

of

observationsk

=

number

of

regressors

in

unrestricted

regressionincluding

a

constant

in

the

unrestricted

regression

(or

the

total

number

ofparameters

to

be

estimated).Understanding

the

F-testF

test

statistic

follows

a

F

distribution.Recall

that

OLS

regression

is

to

minimize

URSS.

If

RRSS

is

NOT

much

higher

than

URSS

the

restriction

issupported

by

the

data

If

RRSS

is

much

higher

than

URSS the

restriction

is

notsupported

by

the

data‘Understanding

the

F-test‘Usually

RRSS>URSS,

so

F-test

statistic

>0F-test

has

two

different

d.f.

(recall

that

T-Test

only

has

one

d.f.=T-K)?is

rejected

if

the

F

test

statistic>the

critical

F-valueH0F-distribution

Table‘Determining

the

Number

of

Restrictions

in

an

F-test‘Examples

:H0:

hypothesisb1

+

b2

=

2b2

=

1

and

b3

=

-1b2

=

0,

b3

=

0

and

b4

=

0No.

of

restrictions,

m123If

the

model

is

yt

=

b1

+

b2x2t

+

b3x3t

+

b4x4t

+

ut,andH0:

b2

=

0,

and

b3

=

0

and

b4

=

0

is

tested

by

the

regression

F-statistic.

Ittests

the

null

hypothesis

that

all

of

the

coefficients

except

the

interceptcoefficient

are

zero.

Note

the

form

of

the

alternative

hypothesis

for

all

tests

when

more

than

onerestriction

is

involved:

H1:

b2

0,

or

b3

0

or

b4

0What

we

Cannot

Test

with

Either

an

F

or

a

t-test‘

We

cannot

test

using

this

framework

hypotheses

which

are

not

linearor

which

are

multiplicative,

e.g.H0:

b2

b3

=

2

or

H0:

b2

2

=

1cannot

be

tested.F-test

Example‘Question:Suppose

a

researcher

wants

to

test

whether

the

returns

on

a

company

stock

(y)show

unit

sensitivity

to

two

factors

(factor

x2

and

factor

x3)

among

threeconsidered.

The

regression

is

carried

out

on

144

monthly

observations.

Theregression

is

yt

=

b1

+

b2x2t

+

b3x3t

+

b4x4t+

utWhat

are

the

restricted

and

unrestricted

regressions?If

the

two

RSS

are

436.1

and

397.2

respectively,

perform

the

test.Goodness

of

Fit:

R2

Goodness

of

fit

measures

how

well

the

model

fits

the

data,

or,

how

well

dothe

X

explain

the

variation

in

Y?Can

RSS

measures

the

goodness

of

fit?

No.

Because

RSS

is

unbounded

from

above

and

it

can

take

any

(non-negative

value

The

value

of

RSS

depends

to

a

great

extent

on

the

scale

of

Y.

e.g.

Y/10

hasless

RSS

than

Y.So

we

need

a

scaled

version

of

RSS:

R2If

this

is

high,

it

meansThe

model

fits

the

data

wellR2

is

the

square

of

the

correlation

coefficient

between

Y

and

y[-1,1][0,1]Goodness

of

Fit:R2‘

For

another

explanation,

recall

that

what

we

are

interested

in

doing

isexplaining

the

variability

of

y

about

its

mean

value,

,

i.e.

the

total

sum

ofsquares,

TSS:ttTSS

y2yGoodness

of

Fit:R2‘

We

can

split

the

TSS

into

two

parts,ESS(explained

sum

of

squares)

andRSS(the

part

which

we

did

not

explain

using

the

model

)Defining

R2That

is,

TSS=

ESS

+

RSSOur

goodness

of

fit

statistic

isR21RSSESS

TSS

RSS

TSS

TSSTSS

R2

must

always

lie

between

zero

and

one.

To

understand

this,

consider

twoextremesRSS

=

TSSi.e.ESS

=

0soR2

=

ESS/TSS

=

0ESS

=

TSSi.e.RSS

=

0soR2

=

ESS/TSS

=

1t

t‘ttttu?2y

y2y?

y2The

Limit

Cases:

R2

=

0

and

R2

=

1ytWhen

R2

=

0

,

the

model

hasno

any

ability

to

explain

the

variationof

Y,

this

would

happen

only

theestimated

coefficients

are

all

zero.yxt‘The

Limit

Cases:

R2

=

0

and

R2

=

1When

R2

=

1

,

the

model

hasexplained

all

of

the

variation

ytof

Y

about

its

mean

value,this

would

happen

only

in

the

casewhen

all

of

the

observations

lie

exactlyon

the

fitted

line.xt‘Problems

with

R2

as

a

Goodness

of

Fit

MeasureThere

are

a

number

of

them:1.R2

is

defined

as

the

variation

about

the

mean

of

y

(

y

)

,

so

if

a

model

isrearranged

and

the

dependent

variable

changes,

R2

will

change.

(ev