.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/model_selection/plot_permutation_tests_for_classification.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end <sphx_glr_download_auto_examples_model_selection_plot_permutation_tests_for_classification.py>` to download the full example code. or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_model_selection_plot_permutation_tests_for_classification.py: ================================================================= 通过排列检验分类评分的显著性 ================================================================= 本示例演示了如何使用 :func:`~sklearn.model_selection.permutation_test_score` 通过排列æ¥è¯„估交å‰éªŒè¯è¯„分的显著性。 .. GENERATED FROM PYTHON SOURCE LINES 9-13 .. code-block:: Python # 作者:scikit-learn å¼€å‘者 # SPDX 许å¯è¯æ ‡è¯†ç¬¦ï¼šBSD-3-Clause .. GENERATED FROM PYTHON SOURCE LINES 14-18 Dataset ------- 我们将使用 :ref:`iris_dataset` ,该数æ®é›†åŒ…å«ä»Ž3ç§é¸¢å°¾èбä¸èŽ·å–的测é‡å€¼ã€‚ .. GENERATED FROM PYTHON SOURCE LINES 18-25 .. code-block:: Python from sklearn.datasets import load_iris iris = load_iris() X = iris.data y = iris.target .. GENERATED FROM PYTHON SOURCE LINES 26-27 我们还将生æˆä¸€äº›éšæœºç‰¹å¾æ•°æ®ï¼ˆå³20个特å¾ï¼‰ï¼Œè¿™äº›ç‰¹å¾ä¸Žé¸¢å°¾èŠ±æ•°æ®é›†ä¸çš„ç±»åˆ«æ ‡ç¾ä¸ç›¸å…³ã€‚ .. GENERATED FROM PYTHON SOURCE LINES 27-36 .. code-block:: Python import numpy as np n_uncorrelated_features = 20 rng = np.random.RandomState(seed=0) # 使用与鸢尾花数æ®é›†ç›¸åŒæ•°é‡çš„æ ·æœ¬å’Œ20ä¸ªç‰¹å¾ X_rand = rng.normal(size=(X.shape[0], n_uncorrelated_features)) .. GENERATED FROM PYTHON SOURCE LINES 37-43 ç½®æ¢æ£€éªŒå¾—分 ---------------------- 接下æ¥ï¼Œæˆ‘们使用原始的鸢尾花数æ®é›†è®¡ç®— :func:`~sklearn.model_selection.permutation_test_score` ,该数æ®é›†å¯ä»¥å¾ˆå¥½åœ°é¢„æµ‹æ ‡ç¾ï¼›åŒæ—¶ä½¿ç”¨éšæœºç”Ÿæˆçš„特å¾å’Œé¸¢å°¾èŠ±æ ‡ç¾è¿›è¡Œè®¡ç®—,这些特å¾å’Œæ ‡ç¾ä¹‹é—´ä¸åº”有任何ä¾èµ–关系。我们使用 :class:`~sklearn.svm.SVC` 分类器和 :ref:`accuracy_score` 在æ¯ä¸€è½®ä¸è¯„估模型。 :func:`~sklearn.model_selection.permutation_test_score` 通过计算分类器在数æ®é›†çš„1000ç§ä¸åŒæŽ’列上的准确率æ¥ç”Ÿæˆä¸€ä¸ªé›¶åˆ†å¸ƒï¼Œå…¶ä¸ç‰¹å¾ä¿æŒä¸å˜ä½†æ ‡ç¾è¿›è¡Œä¸åŒçš„æŽ’列。这是零å‡è®¾çš„分布,零å‡è®¾è®¤ä¸ºç‰¹å¾å’Œæ ‡ç¾ä¹‹é—´æ²¡æœ‰ä¾èµ–关系。然åŽè®¡ç®—一个ç»éªŒp值,å³åœ¨å¤šå°‘百分比的排列ä¸èŽ·å¾—çš„åˆ†æ•°å¤§äºŽä½¿ç”¨åŽŸå§‹æ•°æ®èŽ·å¾—çš„åˆ†æ•°ã€‚ .. GENERATED FROM PYTHON SOURCE LINES 43-58 .. code-block:: Python from sklearn.model_selection import StratifiedKFold, permutation_test_score from sklearn.svm import SVC clf = SVC(kernel="linear", random_state=7) cv = StratifiedKFold(2, shuffle=True, random_state=0) score_iris, perm_scores_iris, pvalue_iris = permutation_test_score( clf, X, y, scoring="accuracy", cv=cv, n_permutations=1000 ) score_rand, perm_scores_rand, pvalue_rand = permutation_test_score( clf, X_rand, y, scoring="accuracy", cv=cv, n_permutations=1000 ) .. GENERATED FROM PYTHON SOURCE LINES 59-63 åŽŸå§‹æ•°æ® ^^^^^^^^^ 䏋颿ˆ‘们绘制了置æ¢å¾—分(原å‡è®¾åˆ†å¸ƒï¼‰çš„直方图。红线表示分类器在原始数æ®ä¸ŠèŽ·å¾—çš„å¾—åˆ†ã€‚è¯¥å¾—åˆ†è¿œé«˜äºŽä½¿ç”¨ç½®æ¢æ•°æ®èŽ·å¾—çš„å¾—åˆ†ï¼Œå› æ¤ p 值éžå¸¸ä½Žã€‚这表明仅å‡å¶ç„¶æ€§è޷得如æ¤é«˜å¾—分的å¯èƒ½æ€§å¾ˆä½Žã€‚è¿™è¯æ˜Žäº†é¸¢å°¾èŠ±æ•°æ®é›†åœ¨ç‰¹å¾å’Œæ ‡ç¾ä¹‹é—´å˜åœ¨çœŸå®žçš„ä¾èµ–关系,并且分类器能够利用这一点æ¥èŽ·å¾—è‰¯å¥½çš„ç»“æžœã€‚ .. GENERATED FROM PYTHON SOURCE LINES 63-75 .. code-block:: Python import matplotlib.pyplot as plt fig, ax = plt.subplots() ax.hist(perm_scores_iris, bins=20, density=True) ax.axvline(score_iris, ls="--", color="r") score_label = f"Score on original\ndata: {score_iris:.2f}\n(p-value: {pvalue_iris:.3f})" ax.text(0.7, 10, score_label, fontsize=12) ax.set_xlabel("Accuracy score") _ = ax.set_ylabel("Probability density") .. image-sg:: /auto_examples/model_selection/images/sphx_glr_plot_permutation_tests_for_classification_001.png :alt: plot permutation tests for classification :srcset: /auto_examples/model_selection/images/sphx_glr_plot_permutation_tests_for_classification_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 76-80 Random data ^^^^^^^^^^^ 䏋颿ˆ‘ä»¬ç»˜åˆ¶äº†éšæœºæ•°æ®çš„零分布。置æ¢å¾—分与使用原始鸢尾花数æ®é›†èŽ·å¾—çš„å¾—åˆ†ç›¸ä¼¼ï¼Œå› ä¸ºç½®æ¢æ€»æ˜¯ç ´å任何å˜åœ¨çš„ç‰¹å¾æ ‡ç¾ä¾èµ–æ€§ã€‚ç„¶è€Œï¼Œåœ¨è¿™ç§æƒ…å†µä¸‹ï¼ŒåŽŸå§‹éšæœºæ•°æ®ä¸ŠèŽ·å¾—çš„å¾—åˆ†éžå¸¸å·®ã€‚这导致了一个很大的p值,è¯å®žäº†åŽŸå§‹æ•°æ®ä¸æ²¡æœ‰ç‰¹å¾æ ‡ç¾ä¾èµ–性。 .. GENERATED FROM PYTHON SOURCE LINES 80-92 .. code-block:: Python fig, ax = plt.subplots() ax.hist(perm_scores_rand, bins=20, density=True) ax.set_xlim(0.13) ax.axvline(score_rand, ls="--", color="r") score_label = f"Score on original\ndata: {score_rand:.2f}\n(p-value: {pvalue_rand:.3f})" ax.text(0.14, 7.5, score_label, fontsize=12) ax.set_xlabel("Accuracy score") ax.set_ylabel("Probability density") plt.show() .. image-sg:: /auto_examples/model_selection/images/sphx_glr_plot_permutation_tests_for_classification_002.png :alt: plot permutation tests for classification :srcset: /auto_examples/model_selection/images/sphx_glr_plot_permutation_tests_for_classification_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 93-103 å¦ä¸€ç§å¯èƒ½å¯¼è‡´é«˜på€¼çš„åŽŸå› æ˜¯åˆ†ç±»å™¨æ— æ³•åˆ©ç”¨æ•°æ®ä¸çš„ç»“æž„ã€‚åœ¨è¿™ç§æƒ…å†µä¸‹ï¼Œåªæœ‰èƒ½å¤Ÿåˆ©ç”¨çŽ°æœ‰ä¾èµ–关系的分类器æ‰ä¼šæœ‰ä½Žp值。在我们上é¢çš„例åä¸ï¼Œç”±äºŽæ•°æ®æ˜¯éšæœºçš„ï¼Œæ‰€æœ‰åˆ†ç±»å™¨çš„på€¼éƒ½ä¼šå¾ˆé«˜ï¼Œå› ä¸ºæ•°æ®ä¸ä¸å˜åœ¨ç»“构。 最åŽè¯·æ³¨æ„,å³ä½¿æ•°æ®ä¸åªæœ‰å¾®å¼±çš„ç»“æž„ï¼Œè¯¥æµ‹è¯•ä¹Ÿå·²è¢«è¯æ˜Žä¼šäº§ç”Ÿè¾ƒä½Žçš„p值[1]_。 .. rubric:: References .. [1] Ojala å’Œ Garriga. `ç”¨äºŽç ”ç©¶åˆ†ç±»å™¨æ€§èƒ½çš„ç½®æ¢æ£€éªŒ <http://www.jmlr.org/papers/volume11/ojala10a/ojala10a.pdf>`_ . 《机器å¦ä¹ ç ”ç©¶æ‚志》(2010) 第11å· .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 6.144 seconds) .. _sphx_glr_download_auto_examples_model_selection_plot_permutation_tests_for_classification.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/scikit-learn/scikit-learn/main?urlpath=lab/tree/notebooks/auto_examples/model_selection/plot_permutation_tests_for_classification.ipynb :alt: Launch binder :width: 150 px .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_permutation_tests_for_classification.ipynb <plot_permutation_tests_for_classification.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_permutation_tests_for_classification.py <plot_permutation_tests_for_classification.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_permutation_tests_for_classification.zip <plot_permutation_tests_for_classification.zip>` .. include:: plot_permutation_tests_for_classification.recommendations .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_