GUIDE teknik

Nested Cross-Validation

Nested cross-validation uses an inner loop to select a model's settings and an outer loop to evaluate the resulting selection procedure on held-out data.

Déglul tegtal biiIA-nettali · ~3 min

Ci xët wii3 simili jàng
  1. Résumé
  2. Plongeur bu xóot
  3. Doxal ci àdduna dëgg
  4. The Future of Nested Cross-Validation
  5. Teg ko ci jëf
  6. Weyal di banneexu
  7. Laaj yi ñuy faral di laaj

Résumé

It reduces the optimistic evaluation that can arise when the same validation scores are used both to choose settings and to claim performance.

Plongeur bu xóot

Searching many model settings is itself a form of learning from data. Even when every candidate uses cross-validation, choosing the candidate with the best score favors settings that benefited from random variation in those validation folds. Reporting that winning score as a fresh performance estimate can therefore be optimistic. Nested cross-validation separates selection from evaluation. Start with an outer split. Set aside its test fold and conduct the entire search using only the outer training data. Inner cross-validation compares the candidate settings. Refit the selected configuration on the outer training data, evaluate it once on the outer test fold, and repeat this process for the other outer folds. The outer scores evaluate a procedure: the model family, search space, preprocessing and selection rule used together. Different outer folds may select different settings. That is expected, because their training data differ. The result is not a competition in which you simply deploy whichever outer-fold model achieved the highest test score. Preprocessing belongs inside the procedure. A scaler, imputer or feature selector fitted on all examples before splitting can leak information. Scikit-learn's Pipeline helps keep these operations attached to model fitting, while GridSearchCV can perform the inner search. The split strategy must also match the intended use. Repeated records from one customer may need group separation. Forecasting requires respect for time order. Nesting an unsuitable random split does not repair that design problem. After evaluating the fixed procedure, perform its selection step on the available development data and fit a final model. Preserve a separate test set if the project uses one. Repeatedly changing the procedure after inspecting outer scores can turn the outer evaluation into another tuning process.

Doxal ci àdduna dëgg

A researcher compares several regularization strengths inside each outer training split. Only the selected model is evaluated on that split's outer test fold.

A hypothetical search uses five outer folds, three inner folds and ten candidate settings. It performs 150 inner candidate fits, plus five refits if the selected model is refitted once per outer fold.

A model uses multiple records from each customer. The team keeps customers separated across both inner and outer splits when it wants to estimate performance on unseen customers.

An analyst places a scikit-learn Pipeline inside GridSearchCV and evaluates that search with an outer cross-validation routine. Preprocessing is learned within the relevant training folds.

The Future of Nested Cross-Validation

As automated model searches become easier to launch, evaluation systems should make the boundary between tuning and testing visible. A useful report records the whole selection recipe and the data it was allowed to inspect. Teams can then compare improvements to a frozen procedure instead of repeatedly optimizing against the same test feedback. Compute limits will still matter, especially for expensive models. The appropriate response is to choose an evaluation design that fits the project and document its limitations, rather than treating extra layers of cross-validation as a universal requirement.

Teg ko ci jëf

Lu tax mu am solo

  • Njëgg ak budget. Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
  • dogal yu gëna leer. Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
  • Xool kalite. Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.

Moytul

  • Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
  • Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
  • Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.

Jéego yi ci topp

  1. Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
  2. Benchmark ci biir sargal ak done yu dëggu.
  3. Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
  4. Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.

Weyal di banneexu

Bulletin bu amul fayda

Jàngal xibaar AI bu bes

Ñatti jaar-jaari IA yuñ firndeel bis bu nekk ci ayu-bis bi, ñu bind leen ci Àngle bu leer. Gratis ba fàww, amul yëgle.

Benn email bis bu nekk ci ayu-bis bi. Benn klik nga dindi abonemaa bi. Dunu musa jaay wala séddoo sa adres.

Nattal sa bopp

Take the Nested Cross-Validation quiz

Feedback ci saasi ci bépp tontu, ak sertifikaa buñ mëna séddoo ak ID buñ mëna saytu soo jàllee ci njàng mi.

Tambalil quiz

Jàppale njàngum IA te doo fay. AI Understanding mbootaay la buy def te Yàlla tax — amul yëgle, amul paywall, musul am. Defal ab sarax

Laaj yi ñuy faral di laaj

What is Nested Cross-Validation?

Nested cross-validation uses an inner loop to select a model's settings and an outer loop to evaluate the resulting selection procedure on held-out data. It reduces the optimistic evaluation that can arise when the same validation scores are used both to choose settings and to claim performance.

Within one outer split, which data may the inner hyperparameter search use?

The outer test fold must remain outside the selection process so it can evaluate the procedure on held-out data.

Why can reporting the best inner validation score overstate model performance?

Selection uses the validation results, so the winning score is not an independent assessment of that selection process.

Five outer folds, three inner folds and ten candidate settings require how many inner candidate fits in the guide's example?

Multiply the outer folds, inner folds and candidates: five times three times ten equals 150.

A scaler is fitted on the full dataset before nested cross-validation begins. Which problem remains?

Nesting does not undo information leakage introduced before splitting. Preprocessing must be fitted within the relevant training folds.

Different outer folds select different hyperparameters. How should this be interpreted?

The outer loop assesses the selection procedure, whose chosen settings may vary with its training sample.