^{1}

^{1}

^{*}

In this paper, we propose a Fast Iteration Method for solving mixture regression problem, which can be treated as a model-based clustering. Compared to the EM algorithm, the proposed method is faster, more flexible and can solve mixture regression problem with different error distributions (
*i.e.* Laplace and t distribution). Extensive numeric experiments show that our proposed method has better performance on randomly simulations and real data.

In some situations, the data may not be suitable for the linear model, such as nonlinear regression, nonparametric regression, generalized linear model. The mixture regression problem discussed in this paper is a situation with mixed data. Specifically, in the observations, some data are from a model, while others are from other models. As in [

where

riable.

The Equation (1.1), a mixture regression model, can also be treated as a model-based clustering [^{−6}).

In [

・ Initialize the value of the parameters:

・ E-Step: At the (k + 1)th iteration, calculate:

・ M-Step: Use the following value:

to maximize:

Solving mixture regression problem based on EM algorithm is a complex work with large amount calculation in each iteration. In this paper, we propose a Fast Iteration Method inspired by k-means clustering [

For the situation of the mixture regression problem, the aim of this problem is to find several linear models (i.e. every parameter β_{k} of the model). While, if we known whether an observation is belong to each population, mixture regression model can be treated as a simple linear model. That is: there exist

Theorem 2.1 The existence of the parameter matrix

Proof

If we know every

If the elements

For the limited combinations, there exists a combination which will lead to the minimum square error.

EM algorithm is meaningful to the mixture regression problem. However, there are still some other methods to solve this question.

The algorithm below is a fast iteration for mixture regression model, which could solve the regression situation with data in different populations. This method is inspired by K-means (the famous clustering algorithm) which calculate the distance between each point to other models and replace the “worst” observation to the suitable model. After finishing this type of calculation for several times, the algorithm will stop until moving any points to other model won’t make the loss-function better, that is, the change of loss function will below a threshold (10^{−6} or 10^{−9}). In the question of small samples, this stop rules will lead to find a best classifying: moving any observation to any other populations will make things worse. Sometimes set a threshold will avoid the algorithm fall into the endless loop.

Our proposed Fast Iteration Method is similar to K-means algorithm, calculate the MSE for each point to every model and change the point which can decrease the MSE most. We summarize our method as follows.

Algorithm:

1) Calculate the initial value: Group information:

parts

2) Get the subset:

3) Fit g linear models from

And get

4.) Calculate:

5)

6) Repeat 2 - 5 until stop

For the method of parameter estimating in the algorithm,

If we need a robust estimation,

In order to validate the rationality of the model, we designed a numeric simulation and generated sample data

Model 1

Model 2

Model 3

For every model considered above, we generated sample using different kinds of distributions: 1) ε~N(0,1); 2) ε~a Laplace distribution with mean 0 and variance 1; 3) ε~0.95N(0,1) + 0.05N(0,25) Mixture Normal distribution; 4) ε~t_{3} t-distribution with degree 3.

We used three methods for comparing. Fast Iteration with Linear Model (FI-OLS), Fast Iteration with median regression model (FI-LAE) and EM algorithm are used for solving mixture regression problem for each model.

Repeat the simulation with 1000 times and we got the bias and MSE of every parameter (see

N (0; 1) | Laplace (1) | Mixture | t_{3} | |

Model 1 1: FI-OLS | ||||

0.0087 (0.2045) | −0.0109 (0.5) | 0.0637623 (14.40) | −0.008 (5.394) | |

0.0234 (0.1407) | −0.0086 (0.2935) | −0.1150 (2.476) | 0.1550 (5.387) | |

0.0029 (0.2004) | 0.0146 (0.5715) | −0.0710 (14.46) | −0.0606 (5.425) | |

−0.0338 (0.1463) | 0.0088 (0.3008) | 0.0863 (2.447) | −0.1359 (4.111) | |

n | 47.447 | 47.374 | 47.654 | 47.491 |

Model 1 1: FI-LAE | ||||

0.0058 (0.1346) | 0.0059 (0.1904) | 0.0690 (9.369) | −0.0029 (0.2625) | |

0.036 (0.1005) | 0.0431 (0.1444) | 0.1024 (2.555) | 0.0326 (0.1920) | |

−0.008 (0.1467) | 0.0026 (0.1861) | −0.0503 (9.676) | −0.0084 (0.2812) | |

−0.037 (0.1034) | −0.0497 (0.1259) | −0.2064 (2.584) | −0.0449 (0.1863) | |

n | 47.772 | 47.505 | 48.505 | 47.795 |

Model 1 1: Mixreg | ||||

−0.0018 (0.0559) | 0.0090 (0.5108) | −0.1052 (12.54) | −0.0306 (2.720) | |

0.0109 (0.0710) | −0.0982 (0.2763) | 0.5713 (4.657) | −0.0309 (2.425) | |

−0.001 (0.0653) | 0.0155 (0.5280) | −0.1060 (12.01) | −0.116 (3.544) | |

−0.0021 (0.0645) | −0.0792 (0.3598) | −0.3928 (3.095) | 0.067 (2.068) |

model 1,

As we can see in the three tables, simulation shows the Fast Iteration for Matrix Regression with LAE performs better in Laplace distribution and t-distribution. More specifically, in

As we described our model as a “fast” iteration method, the FI-OLS and FI-LAE are calculated faster than EM algorithm. For 100 observations with 2 populations, EM got about 0.07 s for mixture regression (Rpackage: Mixreg), while FI-OLS used about 0.02 s (i5, 8G memory).

In the data simulation section, we use the data by Cohen (1984) [

・ Situation 1: Original data.

・ Situation 2: Data with 5 outliers at (3,5).

・ Situation 3: Data with 5 outliers at (1.5,0).

・ Situation 4: Data with 5 outliers at (0,5) (

We used three algorithms in these four situations. In the first situation, the original data is used for regression. In other three situations, 5 outliers in different position are places in the data. (3,5) for the Situation 2, (1.5,0 )for the Situation 3 and (0,5) for Situation 4. The algorithms we used are FI-OLS, FI-LAE and EM algorithm. FI-OLS and FI-LAE are mentioned in our Fast Iteration for Mixture Regression model and the Mixreg package in R [

N (0; 1) | Laplace (1) | Mixture | t_{3} | |

Model 1 2: FI-OLS | ||||

0.067 (0.127) | 0.2335 (0.2649) | 2.596 (8.789) | 0.4682 (3.3621) | |

0.0074 (0.1797) | 3e−04 (0.288) | −0.0584 (3.2318) | 0.0242 (4.3117) | |

−0.0608 (0.1325) | −0.2487 (0.2763) | −2.5688 (8.5827) | −0.4789 (4.7674) | |

0.0243 (0.1991) | −0.031 (0.3553) | 0.0199 (2.698) | 0.0501 (6.8493) | |

n | 48.294 | 48.533 | 48.304 | 48.32 |

Model 1 2: FI-LAE | ||||

−0.0813 (0.1561) | −0.0017 (0.1509) | 1.8594 (5.5725) | 0.0027 (0.2078) | |

−0.0162 (0.2411) | −0.0025 (0.273) | −0.002 (3.2212) | 7e−04 (0.3315) | |

0.0942 (0.1627) | −0.0047 (0.1545) | −1.8665 (5.5066) | 0.0011 (0.1907) | |

0.0284 (0.2515) | −0.0087 (0.2756) | 0.0376 (3.4155) | 0.0122 (0.3787) | |

n | 47.772 | 47.505 | 48.505 | 47.795 |

Model 1 2: Mixreg | ||||

−0.1892 (0.4177) | −0.2054 (1.3923) | 1.3117 (8.5285) | 0.0722 (5.718) | |

8e−04 (0.3108) | −0.0452 (1.4686) | 0.01 (4.8435) | 0.0018 (10.1499) | |

0.1731 (0.4249) | 0.1488 (1.3561) | −1.2019 (7.9067) | 0.093 (3.1488) | |

0.0152 (0.2539) | −0.0631 (1.5643) | −0.0837 (4.3408) | −0.1423 (5.9172) |

N (0; 1) | Laplace (1) | Mixture | t_{3} | |

Model 1 3: FI-OLS | ||||

0.0013 (0.1245) | 0.0328 (0.365) | 0.1437 (11.4582) | 0.1052 (8.0505) | |

−0.0084 (0.1318) | −0.0351 (0.3308) | −0.6633 (4.0265) | −0.222 (4.5141) | |

0.0169 (0.0778) | 0.0244 (0.1954) | 0.1002 (2.3988) | 0.1932 (3.5566) | |

3e−04 (0.1308) | −0.0235 (0.3624) | −0.2424 (11.8562) | 0.0266 (2.0405) | |

0.0128 (0.1179) | 0.0296 (0.31) | 0.7077 (3.9896) | 0.0373 (1.6375) | |

−0.0253 (0.076) | −0.0246 (0.1998) | −0.187 (2.4352) | −0.0791 (1.392) | |

n | 47.457 | 47.348 | 46.584 | 47.053 |

Model 1 3: FI-LAE | ||||

−0.0081 (0.0927) | 0.0067 (0.1161) | 0.0126 (7.4561) | −0.0098 (0.1572) | |

0.0077 (0.0933) | 0.0257 (0.1311) | −0.7597 (4.0911) | 0.0112 (0.1747) | |

0.0068 (0.0726) | 0.028 (0.088) | 0.2371 (2.1575) | 0.0398 (0.1214) | |

0.0042 (0.0949) | −0.0142 (0.1193) | −0.0368 (8.0728) | −0.0102 (0.1622) | |

0.0018 (0.0833) | −0.0176 (0.1295) | 0.7235 (4.0856) | −0.0205 (0.1595) | |

−0.0217 (0.0713) | −0.0389 (0.0949) | −0.2809 (2.1687) | −0.0325 (0.1282) | |

n | 47.353 | 47.674 | 47.457 | 47.333 |

Model 1 3: Mixreg | ||||

0.001 (0.0319) | −8e−04 (0.0914) | 0.0448 (8.1668) | −0.0165 (2.2405) | |

0.0023 (0.0316) | −0.0176 (0.1136) | 0.3945 (2.8743) | 0.0831 (3.1454) | |

0.0029 (0.0353) | −0.0231 (0.1402) | −0.6076 (4.7143) | −0.269 (5.8672) | |

0.0063 (0.0362) | 0.017 (0.1311) | 0.1551 (8.2802) | 0.0256 (1.7082) | |

−0.0098 (0.0401) | 0.0212 (0.1805) | −0.3748 (3.0154) | −0.0387 (1.9867) | |

−0.0017 (0.0349) | 0.0451 (0.183) | 0.6288 (4.9866) | 0.2635 (1.5045) |

In the situation 1, three algorithms got the similar answers. They all perform really well. In other situations, the FI-LAE shows that if there are some outliers in the data, a robust regression will lead to a closer answer, while the EM and FI-OLS affected by the outliers in different degrees.

As the discussion above, we can safely draw the conclusion that the fast iteration for solving mixture regression problem is efficiently and effectively. Compared to ordinary EM algorithm, this method can solve the problem quickly and obtain perfect performance. After changing the method of parameter estimation, the Fast Iteration Method can solve mixture regression when ε is in different distributions.

DaweiLang,WanzhouYe, (2015) A Fast Iteration Method for Mixture Regression Problem. Journal of Applied Mathematics and Physics,03,1100-1107. doi: 10.4236/jamp.2015.39136