Decision Trees and Random Forests

A decision tree uses feature-based tests to route an input toward a leaf prediction. Classification splits commonly seek reductions in Gini impurity or entropy, while regression can use squared-error criteria. A readable path explains the fitted rule followed by an example; it does not establish that a feature has a causal effect. Deep unconstrained trees can fit training-specific details and generalize poorly. A random forest combines many trees with randomized training choices. In a conventional bootstrap forest, each tree receives a sample drawn with replacement from the training rows. Random feature subsets provide another source of variation at splits. The two mechanisms differ: changing sampled rows is not the same as changing the features considered for a split. Combining less-correlated tree predictions can reduce variance, but it does not cancel every mistake or guarantee better accuracy than all alternatives. Classification aggregation may use votes or average class probabilities depending on the implementation; regression commonly averages numeric predictions. Bootstrap sampling also leaves some training rows out of each tree. For a given row, trees that did not train on it can supply an out-of-bag prediction. Aggregated out-of-bag predictions provide an internal generalization estimate without treating those rows as unseen by the entire forest. Gradient boosting is a different ensemble approach that adds trees sequentially according to its training objective. XGBoost is a tree-boosting system, rather than another name for a random forest. Compare these methods using appropriate evaluation conditions and the actual dataset.