Feature Selection Methods

Feature selection keeps some input variables and excludes others. It differs from feature extraction, which transforms inputs into new representations, such as principal components. Selection can reduce measurement or inference costs, simplify a model, and sometimes improve generalization when irrelevant or redundant variables distract the learner. It does not guarantee better accuracy: a discarded variable may contain useful signal, especially through interactions. Filter methods score variables using criteria independent of the final estimator. Examples include variance thresholds and univariate tests or mutual information between one feature and the target. They are often inexpensive, but a univariate score can miss a feature that matters only in combination with another. A correlation filter can also remove one of two redundant inputs without knowing which is more useful to the eventual model. Wrapper methods evaluate candidate subsets by fitting an estimator. Recursive feature elimination repeatedly fits a model, ranks features, and removes some before refitting. This can reflect the estimator's behavior more directly than a simple filter, but repeated fitting costs time and can overfit if selection is judged on the same data used to report performance. Embedded methods perform selection during model fitting. L1-regularized linear models can set coefficients exactly to zero; tree-based methods provide split-based importance measures that can guide selection. These measures have assumptions and biases, so they should not be treated as universal truth. Correlated features can divide importance or substitute for one another. The entire selection procedure belongs inside the training process for evaluation. If feature scores are computed once using all labels before cross-validation, information from validation folds has already influenced the selected subset. This leakage can make measured performance optimistic. Compare the full pipeline, including selection, using appropriate splits. Keep domain constraints, availability at prediction time, fairness, and measurement cost in view alongside scores.