Statistical Methods for the Evaluation of Educational Services and Quality of Products

by Paola Monari (Editor), Matilde Bini (Editor), Domenico Piccolo (Editor), Luigi Salmaso (Editor)

Overview

The book presents statistical methods and models that can usefully support the ev- uation of educational services and quality of products. The contributions collected in this book summarize the work of several researchers from the universities of Bologna, Firenze, Napoli and Padova. The contributions are written with a cons- tent notation and a unified view, and concern methodological advances developed mostly with reference to specific problems of evaluation using real data sets. The evaluation of educational services, as well as the analysis of judgements and preferences, poses severe methodological challenges because of the presence of one or more of the following aspects: the observational (non experimental) nature of the context, which is associated with the well-known problems of selection bias and presence of nuisance factors; the hierarchical structure of the data, that entails c- related observations and consideration of effects at different levels of the hierarchy and their interactions (multilevel analysis); the multivariate and qualitative nature of the dependent variable, that requires the use of ad hoc statistical methodologies; the presence of non observable factors, e. g. the satisfaction, calling for the use of latent variables models; the simultaneous presence of components of pleasure and components of uncertainty in the explication of the judgments, that asks for the specification and estimation of mixture models. The first part of the book deals with latent variable models.

Product Details

ISBN-13:	9783790823844
Publisher:	Physica-Verlag HD
Publication date:	12/14/2009
Series:	Contributions to Statistics
Edition description:	2009
Pages:	243
Product dimensions:	6.20(w) x 9.20(h) x 0.90(d)

Table of Contents

1 Introduction Matilde Bini Paola Monari Domenico Piccolo Luigi Salmaso 1

1.1 Generalized linear latent variable models 1

1.2 Multilevel models 5

1.2.1 Multilevel mixture factor models 6

1.3 Choices and conjoint analysis: critical aspects and recent developments 7

1.4 Robust diagnostic analysis with forward search 8

1.5 Nonparametric combination of dependent permutation tests and rankings 10

1.5.1 Introduction to permutation tests 11

1.5.2 Multivariate permutation tests and nonparametric combination methodology 12

1.5.3 Nonparametric combination of dependent rankings 14

2 Latent variable models for ordinal data Silvia Cagnone Stefania Mignani Irini Moustaki 17

2.1 Introduction 17

2.2 The GLLVM for ordinal data 18

2.2.1 Model specification 18

2.2.2 Model estimation 19

2.3 The goodness-of-fit of the model 21

2.3.1 The problem of sparseness 21

2.3.2 An overall goodness-of-fit test 23

2.4 GLLVM for longitudinal ordinal data 24

2.5 Case study: perceptions of prejudice on American campus 25

2.6 Concluding remarks 28

3 Issues on item response theory modelling Mariagiulia Matteucci Stefania Mignani Bernard P. Veldkamp 29

3.1 Introduction 29

3.2 Basics of item response theory 30

3.2.1 Parameter estimation 32

3.3 Advances in IRT: some issues 36

3.3.1 Multidimensionality 37

3.3.2 Incomplete design 40

3.3.3 Inclusion of prior information 41

3.4 Case study: prior information in educational assessment 43

3.5 Concluding remarks 44

4 Nonlinearity in the analysis of longitudinal data Estela Bee Dagum Silvia Bianconcini Paola Monari 47

4.1 Introduction 47

4.20 Latent curve models 48

4.2.1 Estimation 50

4.3 Modelling nonlinearity in latent curve models 50

4.3.1 Polynomial trajectories 51

4.3.2 Exponential trajectories 52

4.3.3 Complex nonlinear curves 53

4.4 Case study: analysis of university student achievements 55

4.4.1 The data 55

4.4.2 Results 57

4.5 Concluding remarks 60

5 Multilevel models for the evaluation of educational institutions: a review Leonardo Grilli Carla Rampichini 61

5.1 The evaluation of educational institutions 61

5.2 Effectiveness 62

5.2.1 The value-added approach 64

5.2.2 Type A and type B effectiveness 64

5.3 Multilevel models as a tool for measuring effectiveness 65

5.3.1 The random intercept model 66

5.3.2 The random slope model 69

5.3.3 Cross-level interactions 69

5.3.4 Fixed versus random effects 70

5.3.5 Non-linear and multivariate multilevel models 71

5.3.6 Multilevel models for non-hierarchical structures 72

5.4 Issues in model specification 72

5.4.1 Simple versus complex models 72

5.4.2 To adjust or not to adjust? 73

5.4.3 Endogeneity 74

5.4.4 Modelling the achievement progress 75

5.4.5 Measurement error 76

5.5 Use of the model results 77

5.5.1 Ranking the schools 77

5.5.2 Predicting the outcome 79

5.6 Concluding remarks 79

6 Multilevel mixture factor models for the evaluation of educational programs' effectiveness Roberta Varriale Caterina Giusti 81

6.1 Introduction 81

6.2 The multilevel mixture factor model 82

6.3 The Generalized Latent Variable framework 85

6.4 Likelihood, estimation and posterior analysis 87

6.5 Model selection 90

6.6 Case study 91

7 A class of statistical models for evaluating services and performances Marcella Corduas Maria Iannario Domenico Piccolo 99

7.1 Introduction 99

7.2 Unobserved components in the evaluation process 100

7.2.1 Rationale for a new class of models 101

7.3 Specification and properties of CUB models 102

7.4 Inferential issues and numerical procedures 106

7.4.1 The EM algorithm 106

7.4.2 Fitting measures 109

7.5 Fields of application 110

7.6 Further developments: a clustering approach 111

7.7 Case study 112

7.8 Concluding remarks 117

8 Choices and conjoint analysis: critical aspects and recent developments Rossella Berni Riccardo Rivello 119

8.1 Introduction 119

8.2 Literature review 120

8.2.1 Choice Experiment: theory and advances 123

8.2.2 Conjoint Analysis: theory and advances 128

8.3 Our proposal: conjoint analysis and response surface methodology 130

8.3.1 The outlined theory 130

8.3.2 The searching of the best profile through optimization 132

8.4 Case study 133

8.4.1 Optimization results 134

8.5 Concluding remarks 137

9 Robust diagnostics in university performance studies Matilde Bini Bruno Bertaccini Silvia Bacci 139

9.1 Introduction 139

9.2 Robust methods vs diagnostic analysis 142

9.2.1 The Forward Search algorithm 145

9.3 The Forward Search for Generalized Linear Models 146

9.3.1 Robust GLMs for the university effectiveness evaluation. The case of the first year college drop out rate 147

9.4 The Forward Search for ANOVA models 153

9.4.1 The Forward Search for the fixed effects ANOVA 154

9.4.2 The Forward Search for the random effects ANOVA 156

9.4.3 The use of the robust ANOVA for the evaluation of the Italian university reform 158

9.5 Concluding remarks 159

10 A novel global performance score with an application to the evaluation of new detergents Stefano Bonnini Livio Corain Antonio Cordellina Anna Crestana Remigio Musci Luigi Salmaso 161

10.1 Introduction 161

10.2 Composite indexes 163

10.2.1 Standardization: data transformations to obtain homogeneous variables 163

10.2.2 Aggregation: synthesis of information 165

10.3 Global performance score 167

10.3.1 Global score on primary performance 167

10.3.2 Global score on secondary performance 169

10.3.3 Aggregation of GSP and GSS 170

10.4 Case study: comparative performance evaluations of new detergents 170

10.5 A comparative simulation study 173

10.6 Conclusions 176

11 Nonparametric tests for the randomized complete block design with ordered categorical variables Livio Corain Luigi Salmaso 181

11.1 Introduction 181

11.2 Overview on procedures proposed in the literature for the RCB design 182

11.3 Permutation tests for multivariate RCB design 185

11.4 Simulation study 188

11.5 Case study 190

11.6 Conclusions 192

12 A permutation test for umbrella alternatives Dario Basso Fortunato Pesarin Luigi Salmaso 193

12.1 Introduction 193

12.2 Simple stochastic ordering alternatives 196

12.3 Permutation test for umbrella alternatives 199

12.4 Simulation study 201

12.5 Case study: graduates in engineering 204

13 Nonparametric methods for measuring concordance between rankings: a case study on the evaluation of professional profiles of municipal directors Rosa Arboretti Giancristofaro Mario Bolzan Livio Corain 209

13.1 Introduction 209

13.2 Sample survey of municipal directors' professional profiles 210

13.2.1 Context of the evaluation of the role of Communes and of municipal directors 210

13.2.2 Sample survey among Communes of the Veneto 211

13.3 Analysis of concordance between rankings 212

13.3.1 The construction of ranks 212

13.3.2 Hypothesis testing on concordance between rankings 213

13.3.3 Closed testing procedure 216

13.3.4 Rankings of the municipal director's professional profile in the Communes of the Veneto survey 219

13.3.5 Discussion 220

References 227

Index 241