This is a real issue for BIC. Nevertheless, there are a lot of researchers who say BIC is better than AIC, using model recovery simulations as an argument. These simulations consist of generating data from models A and B, and then fitting both datasets with the two models. Overfitting occurs when the wrong model fits the data better than the generating.
Usually, the results point to the fact that AIC is too liberal and still frequently prefers a more complex, wrong model over a simpler, true model. At first glance these simulations seem to be really good arguments, but the problem with them is that they are meaningless for AIC. As I said before, AIC does not consider that any of the candidate models being tested is actually true. According to AIC, all models are approximations to reality, and reality should never have a low dimensionality.
At least lower than some of the candidate models. Most of the times they will agree on the preferred model, when they don't, just report it.
There are several measures stemming from MDL, like normalized maximum likelihood or the Fisher Information approximation. Still, if you want to stick to simple solutions, a nice way for assessing model flexibility especially when the number of parameters are equal, rendering AIC and BIC useless is doing Parametric Bootstrap, which is quite easy to implement.
Here is a link to a paper on it. Some people here advocate for the use of cross-validation. I personally have used it and don't have anything against it, but the issue with it is that the choice among the sample-cutting rule leave-one-out, K-fold, etc is an unprincipled one. Though AIC and BIC are both Maximum Likelihood estimate driven and penalize free parameters in an effort to combat overfitting, they do so in ways that result in significantly different behavior.
Lets look at one commonly presented version of the methods which results form stipulating normally distributed errors and other well behaving assumptions :. The best model in the group compared is the one that minimizes these scores, in both cases.
Clearly, AIC does not depend directly on sample size. Diachronically, as data is introduced and the scores are recalculated, at relatively low N 7 and less BIC is more tolerant of free parameters than AIC, but less tolerant at higher N as the natural log of N overcomes 2.
Additionally, AIC is aimed at finding the best approximating model to the unknown data generating process via minimizing expected estimated K-L divergence. As such, it fails to converge in probability to the true model assuming one is present in the group evaluated , whereas BIC does converge as N tends to infinity.
So, as in many methodological questions, which is to be preferred depends upon what you are trying to do, what other methods are available, and whether or not any of the features outlined convergence, relative tolerance for free parameters, minimizing expected K-L divergence , speak to your goals. In my experience, BIC results in serious underfitting and AIC typically performs well, when the goal is to maximize predictive discrimination.
Ripley provides some remarks on the assumptions behind the mathematical results. Contrary to what some of the other answers indicate, Ripley emphasizes that AIC is based on assuming that the model is true. If the model is not true, a general computation will reveal that the "number of parameters" has to be replaced by a more complicated quantity.
Some references are given in Ripleys slides. Note, however, that for linear regression strictly speaking with a known variance the, in general, more complicated quantity simplifies to be equal to the number of parameters. I would say that while both are quite weak in comparison to for instance cross-validation it is better to use AIC, than more people will be familiar with the abbreviation -- indeed I have never seen a paper or a program where BIC would be used still I admit that I'm biased to problems where such criteria simply don't work.
Edit: AIC and BIC are equivalent to cross-validation provided two important assumptions -- when they are defined, so when the model is a maximum likelihood one and when you are only interested in model performance on a training data. In case of collapsing some data into some kind of consensus they are perfectly ok.
In case of making a prediction machine for some real-world problem the first is false, since your training set represent only a scrap of information about the problem you are dealing with, so you just can't optimize your model; the second is false, because you expect that your model will handle the new data for which you can't even expect that the training set will be representative. And to this end CV was invented; to simulate the behavior of the model when confronted with an independent data.
In case of model selection, CV gives you not only the quality approximate, but also quality approximation distribution, so it has this great advantage that it can say "I don't know, whatever the new data will come, either of them can be better. A penalty function is used in these methods, which is a function of the number of parameters in the model. When n is large the two models will produce quite different results. However, as stated in Wikipedia on BIC :.
AIC should rarely be used, as it is really only valid asymptotically. The main exception to using AICc is when the underlying distributions are heavily leptokurtic. They are both mathematically convenient approximations one can make in order to efficiently compare models.
If they give you different "best" models, it probably means you have high model uncertainty, which is more important to worry about than whether you should use AIC or BIC. I personally like BIC better because it asks more less of a model if it has more less data to fit its parameters - kind of like a teacher asking for a higher lower standard of performance if their student has more less time to learn about the subject. To me this just seems like the intuitive thing to do.
But then I am certain there also exists equally intuitive and compelling arguments for AIC as well, given its simple form. Now any time you make an approximation, there will surely be some conditions when those approximations are rubbish. This can be seen certainly for AIC, where there exist many "adjustments" AICc to account for certain conditions which make the original approximation bad.
This is also present for BIC, because various other more exact but still efficient methods exist, such as Fully Laplace Approximations to mixtures of Zellner's g-priors BIC is an approximation to the Laplace approximation method for integrals.
One place where they are both crap is when you have substantial prior information about the parameters within any given model. AIC and BIC unnecessarily penalise models where parameters are partially known compared to models which require parameters to be estimated from the data. So from a logical viewpoint, any proposition which would lead one to BIC as an approximation are equally supported by the data.
Summary 1. Author Recent Posts. Prabhat S. Latest posts by Prabhat S see all. Help us improve. Rate this post! Cancel Reply. Follow Us. It serves the most optimal coverage of assumptions. Its penalty terms are smaller. Many researchers believe it benefits with the minimum risks while presuming. Because here, n is larger than k 2.
Bayesian Information Criteria BIC is an evaluation of the purpose of the possibility, following the model is accurate, under a particular Bayesian structure. So a lower BIC means that a model is acknowledged to be further anticipated to be the precise model. The theory was developed and published by Gideon E. Schwarz in the year To reach a true-model, it requires probability exactly 1. For false-positive outcomes, it is helpful.
The penalty terms are substantial. Its dimension is finite that gives consistent and easy results. Scientists say that its optimal coverage is less than AIC for assumptions. That even sequences into maximum risk-taking.
0コメント