{"id":9426,"date":"2025-06-11T12:46:02","date_gmt":"2025-06-11T04:46:02","guid":{"rendered":"\/?p=9426"},"modified":"2025-06-11T12:46:46","modified_gmt":"2025-06-11T04:46:46","slug":"%e4%b8%89%e7%a7%8d%e8%af%8d%e8%a2%8b%e6%a8%a1%e5%9e%8bcountvectorizertfidfhashvectorizer","status":"publish","type":"post","link":"\/?p=9426","title":{"rendered":"\u4e09\u79cd\u8bcd\u888b\u6a21\u578bCountVectorizerTFIDFHashVectorizer"},"content":{"rendered":"

1. CountVectorizer<\/strong><\/h3>\n

CountVectorizer<\/code>\u7c7b\u4f1a\u5c06\u6587\u672c\u4e2d\u7684\u8bcd\u8bed\u8f6c\u6362\u4e3a\u8bcd\u9891\u77e9\u9635\u3002 \u4f8b\u5982\u77e9\u9635\u4e2d\u5305\u542b\u4e00\u4e2a\u5143\u7d20a[i][j]\uff0c\u5b83\u8868\u793aj\u8bcd\u5728i\u7c7b\u6587\u672c\u4e0b\u7684\u8bcd\u9891\u3002\u5b83\u901a\u8fc7fit_transform<\/code>\u51fd\u6570\u8ba1\u7b97\u5404\u4e2a\u8bcd\u8bed\u51fa\u73b0\u7684\u6b21\u6570\uff0c\u901a\u8fc7get_feature_names()<\/code>\u53ef\u83b7\u53d6\u8bcd\u888b\u4e2d\u6240\u6709\u6587\u672c\u7684\u5173\u952e\u5b57\uff0c\u901a\u8fc7toarray()<\/code>\u53ef\u770b\u5230\u8bcd\u9891\u77e9\u9635\u7684\u7ed3\u679c\u3002<\/p>\n

from sklearn.feature_extraction.text import CountVectorizer\n#\u8bed\u6599\ncorpus = [\n    'This is the first document.',\n    'This is the this second second document.',\n    'And the third one.',\n    'Is this the first document?'\n]\n#\u5c06\u6587\u672c\u4e2d\u7684\u8bcd\u8f6c\u6362\u6210\u8bcd\u9891\u77e9\u9635\nvectorizer = CountVectorizer()\nprint(vectorizer)\n#\u8ba1\u7b97\u67d0\u4e2a\u8bcd\u51fa\u73b0\u7684\u6b21\u6570\nX = vectorizer.fit_transform(corpus)\nprint(type(X),X)\n#\u83b7\u53d6\u8bcd\u888b\u4e2d\u6240\u6709\u6587\u672c\u5173\u952e\u8bcd\nword = vectorizer.get_feature_names()\nprint(word)\n#\u67e5\u770b\u8bcd\u9891\u7ed3\u679c\nprint(X.toarray())<\/code><\/pre>\n
\u7ed3\u679c\uff1a<\/p>\n
<class 'scipy.sparse._csr.csr_matrix'>   (np.int32(0), np.int32(8)) 1\n  (np.int32(0), np.int32(3))    1\n  (np.int32(0), np.int32(6))    1\n  (np.int32(0), np.int32(2))    1\n  (np.int32(0), np.int32(1))    1\n  (np.int32(1), np.int32(8))    2\n  (np.int32(1), np.int32(3))    1\n  (np.int32(1), np.int32(6))    1\n  (np.int32(1), np.int32(1))    1\n  (np.int32(1), np.int32(5))    2\n  (np.int32(2), np.int32(6))    1\n  (np.int32(2), np.int32(0))    1\n  (np.int32(2), np.int32(7))    1\n  (np.int32(2), np.int32(4))    1\n  (np.int32(3), np.int32(8))    1\n  (np.int32(3), np.int32(3))    1\n  (np.int32(3), np.int32(6))    1\n  (np.int32(3), np.int32(2))    1\n  (np.int32(3), np.int32(1))    1\n['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']\n[[0 1 1 1 0 0 1 0 1]\n [0 1 0 1 0 2 1 0 2]\n [1 0 0 0 1 0 1 1 0]\n [0 1 1 1 0 0 1 0 1]]<\/code><\/pre>\n
\u6211\u4eec\u770b\u4e0b\u8bcd\u7684\u5206\u5e03\uff1a<\/p>\n\n\n\n\n\n\n\n\n
and<\/th>\ndocument<\/th>\nfirst<\/th>\nis<\/th>\none<\/th>\nsecond<\/th>\nthe<\/th>\nthird<\/th>\nthis<\/th>\n<\/tr>\n<\/thead>\n
0<\/td>\n1<\/td>\n1<\/td>\n1<\/td>\n0<\/td>\n0<\/td>\n1<\/td>\n0<\/td>\n1<\/td>\n<\/tr>\n
0<\/td>\n1<\/td>\n0<\/td>\n1<\/td>\n0<\/td>\n2<\/td>\n1<\/td>\n0<\/td>\n2<\/td>\n<\/tr>\n
1<\/td>\n0<\/td>\n0<\/td>\n0<\/td>\n1<\/td>\n0<\/td>\n1<\/td>\n1<\/td>\n0<\/td>\n<\/tr>\n
0<\/td>\n1<\/td>\n1<\/td>\n1<\/td>\n0<\/td>\n0<\/td>\n1<\/td>\n0<\/td>\n1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\u518d\u770b\u4e0b\u539f\u6bb5\u843d\uff0c\u53ef\u4ee5\u9a8c\u8bc1\u4ee5\u4e0b\u5206\u5e03\u662f\u5426\u6b63\u786e(a[i][j]\uff0c\u5b83\u8868\u793aj\u8bcd\u5728i\u7c7b\u6587\u672c\u4e0b\u7684\u8bcd\u9891<\/em><\/u>)\uff1a<\/p>\n
1:    'This is the first document.',\n2:    'This is the this second second document.',\n3:    'And the third one.',\n4:    'Is this the first document?'<\/code><\/pre>\n
\u5173\u4e8e\u7ed3\u679c X\uff1a<\/h4>\n
X = vectorizer.fit_transform(corpus)<\/code>:<\/p>\n
\"image-20250611113043782\"<\/p>\n
\u6587\u6863\u8bcd\u9891\u77e9\u9635\uff08\u7a20\u5bc6\u5f62\u5f0f\uff09<\/strong>\uff1a<\/p>\n\n\n\n\n\n\n\n\n\n
\u6587\u6863<\/th>\nand<\/th>\ndocument<\/th>\nfirst<\/th>\nis<\/th>\none<\/th>\nsecond<\/th>\nthe<\/th>\nthird<\/th>\nthis<\/th>\n\u975e\u96f6\u503c\u6570\u91cf<\/strong><\/th>\n<\/tr>\n<\/thead>\n
1<\/td>\n0<\/td>\n1<\/td>\n1<\/td>\n1<\/td>\n0<\/td>\n0<\/td>\n1<\/td>\n0<\/td>\n1<\/td>\n5\u4e2a<\/strong> (1+1+1+1+1)<\/td>\n<\/tr>\n
2<\/td>\n0<\/td>\n1<\/td>\n0<\/td>\n1<\/td>\n0<\/td>\n2<\/td>\n1<\/td>\n0<\/td>\n2<\/td>\n6\u4e2a<\/strong> (1+1+2+1+2)<\/td>\n<\/tr>\n
3<\/td>\n1<\/td>\n0<\/td>\n0<\/td>\n0<\/td>\n1<\/td>\n0<\/td>\n1<\/td>\n1<\/td>\n0<\/td>\n4\u4e2a<\/strong> (1+1+1+1)<\/td>\n<\/tr>\n
4<\/td>\n0<\/td>\n1<\/td>\n1<\/td>\n1<\/td>\n0<\/td>\n0<\/td>\n1<\/td>\n0<\/td>\n1<\/td>\n4\u4e2a<\/strong> (1+1+1+1+1)<\/td>\n<\/tr>\n
\u603b\u8ba1<\/strong><\/td>\n-<\/td>\n-<\/td>\n-<\/td>\n-<\/td>\n-<\/td>\n-<\/td>\n-<\/td>\n-<\/td>\n-<\/td>\n19\u4e2a\u975e\u96f6\u503c<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\u77e9\u9635\u7ef4\u5ea6\u662f4(\u6587\u6863\u6570)\u00d79(\u8bcd\u6c47\u91cf)=36\uff0c\u4f46\u5b9e\u9645\u975e\u96f6\u5143\u7d20\u53ea\u670919\u4e2a\uff08\u89c1X.data\u6570\u7ec4\u957f\u5ea6\uff09\uff0c\u56e0\u6b64X.size=19\u53cd\u6620\u7684\u662f\u6240\u6709\u6587\u6863\u4e2d\u5b9e\u9645\u51fa\u73b0\u7684\u8bcd\u6c47\u8ba1\u6570\u603b\u548c<\/p>\n
\u5173\u4e8e\u8bcd\u6c47\u8868\u987a\u5e8f\uff1a<\/h4>\n
vectorizer.get_feature_names()<\/code> \u8fd4\u56de\u7684\u8bcd\u6c47\u5217\u8868\u6309 \u5b57\u6bcd\u5347\u5e8f<\/strong> \u6392\u5217\uff08\u4ece a \u5230 z\uff09\uff0c\u8fd9\u662f CountVectorizer<\/code> \u7684\u9ed8\u8ba4\u884c\u4e3a\u3002
\n\u5728\u7528\u6237\u4ee3\u7801\u7684\u8bed\u6599\u4e2d\uff0c\u8bcd\u6c47\u987a\u5e8f\u4e3a\uff1a
\n['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']<\/code><\/p>\n