TermExtract::BrillsTagger
-- 蟆る摩逕ｨ隱樊歓蜃ｺ繝｢繧ｸ繝･繝ｼ繝ｫ��"Brill's Tagger"迚�)
use TermExtract::BrillsTagger;
蜈･蜉帙ユ繧ｭ繧ｹ繝医ｒ縲�"Brill's Tagger"�郁恭譁��蜩∬ｩ槭ち繧ｰ莉倅ｸ弱�繝ｭ繧ｰ繝ｩ繝��峨↓縺�
縺代√◎縺ｮ邨先棡繧偵ｂ縺ｨ縺ｫ蜈･蜉帙ユ繧ｭ繧ｹ繝医°繧牙ｰる摩逕ｨ隱槭ｒ謚ｽ蜃ｺ縺吶ｋ繝励Ο繧ｰ繝ｩ繝�縲�
Brill's Tagger繧貞�縺ｫ縺励※菴懊ｉ繧後◆ Monty Tagger 縺ｫ繧ょｯｾ蠢懊＠縺ｦ縺�ｋ縲�
縺ｪ縺翫。rill's Tagger 縺ｧ繧ｿ繧ｰ莉倥￠繧定｡後≧蝣ｴ蜷医�縲∽ｺ句燕縺ｫBrill's Tagger莉�
螻槭�Perl繧ｹ繧ｯ繝ｪ繝励ヨ Tokenizer.pl 繧偵°縺代※縺翫￥縺薙→繧呈耳螂ｨ縺励※縺�ｋ縲�
蠖薙Δ繧ｸ繝･繝ｼ繝ｫ縺ｮ菴ｿ逕ｨ豕輔↓縺､縺�※縺ｯ縲∬ｦｪ繧ｯ繝ｩ繧ｹ��TermExtract::Calc_Imp)縺九�
莉･荳九�繧ｵ繝ｳ繝励Ν繧ｹ繧ｯ繝ｪ繝励ヨ繧貞盾辣ｧ縺ｮ縺薙→縲�
#!/usr/local/bin/perl -w
#
# ex_BT.pl
#
#縲繝輔ぃ繧､繝ｫ縺九ｉBrill's Tagger 縺ｮ蜃ｦ逅�ｵ先棡繧定ｪｭ縺ｿ蜿悶ｊ
# 讓呎ｺ門�蜉帙↓蟆る摩逕ｨ隱槭→縺昴�驥崎ｦ∝ｺｦ繧定ｿ斐☆繝励Ο繧ｰ繝ｩ繝�
#
# version 0.14
#
#
use TermExtract::BrillsTagger;
#use strict;
my $data = new TermExtract::BrillsTagger;
my $InputFile = "BT_out.txt"; # 蜈･蜉帙ヵ繧｡繧､繝ｫ謖�ｮ�
# 繝励Ο繧ｻ繧ｹ縺ｮ逡ｰ蟶ｸ邨ゆｺ�凾蜃ｦ逅�
# (繝ｭ繝�け繝�ぅ繝ｬ繧ｯ繝医Μ繧剃ｽｿ逕ｨ縺励◆蝣ｴ蜷医�縺ｿ��
$SIG{INT} = $SIG{QUIT} = $SIG{TERM} = 'sigexit';
# 蜃ｺ蜉帙Δ繝ｼ繝峨ｒ謖�ｮ�
# 1 竊� 蟆る摩逕ｨ隱橸ｼ矩㍾隕∝ｺｦ縲�2 竊� 蟆る摩逕ｨ隱槭�縺ｿ
# 3 竊� 繧ｫ繝ｳ繝槫玄蛻�ｊ
my $output_mode = 1;
#
# 驥崎ｦ∝ｺｦ險育ｮ励〒縲�｣謗･隱槭�"蟒ｶ縺ｹ謨ｰ"縲�"逡ｰ縺ｪ繧頑焚"縲�"繝代�繝励Ξ繧ｭ繧ｷ繝�ぅ"縺ｮ縺�
# 縺壹ｌ繧偵→繧九°驕ｸ謚槭ゅヱ繝ｼ繝励Ξ繧ｭ繧ｷ繝�ぅ縺ｯ縲悟ｭｦ鄙呈ｩ溯�縲阪ｒ菴ｿ縺医↑縺�
# 縺ｾ縺溘�"騾｣謗･隱槭�諠��ｱ繧剃ｽｿ繧上↑縺�"驕ｸ謚槭ｂ縺ゅｊ縲√％縺ｮ蝣ｴ蜷医�逕ｨ隱槫�迴ｾ蝗樊焚
# (縺ｨ險ｭ螳壹＆繧後※縺�ｌ縺ｰIDF縺ｮ邨�∩蜷医ｏ縺幢ｼ峨〒驥崎ｦ∝ｺｦ險育ｮ励ｒ陦後≧
# �医ョ繝輔か繝ｫ繝医�"蟒ｶ縺ｹ謨ｰ"繧偵→繧� $obj->use_total)
#
#$data->use_total; # 蟒ｶ縺ｹ謨ｰ繧偵→繧�
#$data->use_uniq; # 逡ｰ縺ｪ繧頑焚繧偵→繧�
#$data->use_Perplexity; # 繝代�繝励Ξ繧ｭ繧ｷ繝�ぅ繧偵→繧�(TermExtract 3.04 莉･荳�)
#$data->no_LR; # 髫｣謗･諠��ｱ繧剃ｽｿ繧上↑縺� (TermExtract 4.02 莉･荳�)
#
# 驥崎ｦ∝ｺｦ險育ｮ励〒縲�｣謗･諠��ｱ縺ｫ謗帙￠蜷医ｏ縺帙ｋ逕ｨ隱槫�迴ｾ鬆ｻ蠎ｦ諠��ｱ繧帝∈謚槭☆繧�
# $data->no_LR; 縺ｨ縺ｮ邨�∩蜷医ｏ縺帙〒逕ｨ隱槫�迴ｾ鬆ｻ蠎ｦ縺ｮ縺ｿ縺ｮ驥崎ｦ∝ｺｦ繧らｮ怜�蜿ｯ閭ｽ
# �医ョ繝輔か繝ｫ繝医� "Frequency" $data->use_frq)
# TF縺ｯ縺ゅｋ逕ｨ隱槭′莉悶�逕ｨ隱槭�荳驛ｨ縺ｫ菴ｿ繧上ｌ縺ｦ縺�◆蝣ｴ蜷医↓繧ゅき繧ｦ繝ｳ繝�
# Frequency 縺ｯ逕ｨ隱槭′莉悶�逕ｨ隱槭�荳驛ｨ縺ｫ菴ｿ繧上ｌ縺ｦ縺�◆蝣ｴ蜷医↓繧ｫ繧ｦ繝ｳ繝医＠縺ｪ縺�
#
#$data->use_TF; # TF (Term Frequency) (TermExtract 4.02 莉･荳�)
#$data->use_frq; # Frequency縺ｫ繧医ｋ逕ｨ隱樣�ｻ蠎ｦ
#$data->no_frq; # 鬆ｻ蠎ｦ諠��ｱ繧剃ｽｿ繧上↑縺�
#
# 驥崎ｦ∝ｺｦ險育ｮ励〒縲∝ｭｦ鄙呈ｩ溯�繧剃ｽｿ縺�°縺ｩ縺�°驕ｸ謚�
# �医ョ繝輔か繝ｫ繝医�縲∽ｽｿ逕ｨ縺励↑縺� $obj->no_stat)
#
#$data->use_stat; # 蟄ｦ鄙呈ｩ溯�繧剃ｽｿ縺�
#$data->no_stat; # 蟄ｦ鄙呈ｩ溯�繧剃ｽｿ繧上↑縺�
#
# 驥崎ｦ∝ｺｦ險育ｮ励〒縲√後ラ繧ｭ繝･繝｡繝ｳ繝井ｸｭ縺ｮ逕ｨ隱槭�鬆ｻ蠎ｦ縲阪→縲碁｣謗･隱槭�驥崎ｦ∝ｺｦ縲�
# 縺ｮ縺ｩ縺｡繧峨↓豈秘㍾繧偵♀縺上°繧定ｨｭ螳壹☆繧九�
# 繝�ヵ繧ｩ繝ｫ繝亥､縺ｯ��
# 蛟､縺悟､ｧ縺阪＞縺ｻ縺ｩ縲後ラ繧ｭ繝･繝｡繝ｳ繝井ｸｭ縺ｮ逕ｨ隱槭�鬆ｻ蠎ｦ縲阪�豈秘㍾縺碁ｫ倥∪繧�
#
#$data->average_rate(0.5);
#
# 蟄ｦ鄙呈ｩ溯�逕ｨDB縺ｫ繝��繧ｿ繧定塘遨阪☆繧九°縺ｩ縺�°驕ｸ謚�
# 驥崎ｦ∝ｺｦ險育ｮ励〒縲∝ｭｦ鄙呈ｩ溯�繧剃ｽｿ縺�→縺阪�縲√そ繝�ヨ縺励※縺翫＞縺溘⊇縺�′
# 辟｡髮｣縲ょ�逅�ｯｾ雎｡縺ｫ蟄ｦ鄙呈ｩ溯�逕ｨDB縺ｫ逋ｻ骭ｲ縺輔ｌ縺ｦ縺�↑縺�ｪ槭′蜷ｫ縺ｾ繧後ｋ
# 縺ｨ豁｣縺励￥蜍穂ｽ懊＠縺ｪ縺��
# �医ョ繝輔か繝ｫ繝医�縲∬塘遨阪＠縺ｪ縺� $obj->no_storage��
#
#$data->use_storage; # 闢�ｩ阪☆繧�
#$data->no_storage; # 闢�ｩ阪＠縺ｪ縺�
#
# 蟄ｦ鄙呈ｩ溯�逕ｨDB縺ｫ菴ｿ逕ｨ縺吶ｋDBM繧担DBM_File縺ｫ謖�ｮ�
# �医ョ繝輔か繝ｫ繝医�縲．B_File縺ｮBTREE繝｢繝ｼ繝会ｼ�
#
#$data->use_SDBM;
#
# 驕主悉縺ｮ繝峨く繝･繝｡繝ｳ繝医�邏ｯ遨咲ｵｱ險医ｒ菴ｿ縺��ｴ蜷医�繝��繧ｿ繝吶�繧ｹ縺ｮ
# 繝輔ぃ繧､繝ｫ蜷阪ｒ繧ｻ繝�ヨ
# �医ョ繝輔か繝ｫ繝医� "stat.db"縺ｨ"comb.db"��
#
#$data->stat_db("stat.db");
#$data->comb_db("comb.db");
#
# 繝��繧ｿ繝吶�繧ｹ縺ｮ謗剃ｻ悶Ο繝�け縺ｮ縺溘ａ縺ｮ荳譎ゅョ繧｣繝ｬ繧ｯ繝医Μ繧呈欠螳�
# 繝�ぅ繝ｬ繧ｯ繝医Μ蜷阪′遨ｺ譁�ｭ怜��医ョ繝輔か繝ｫ繝茨ｼ峨�蝣ｴ蜷医�繝ｭ繝�け縺励↑縺�
#
#$data->lock_dir("lock_dir");
#
# 蜩∬ｩ槭ち繧ｰ莉倥￠貂医∩縺ｮ繝�く繧ｹ繝医°繧峨√ョ繝ｼ繧ｿ繧定ｪｭ縺ｿ霎ｼ縺ｿ
# 蟆る摩逕ｨ隱槭Μ繧ｹ繝医ｒ驟榊�縺ｫ霑斐☆
# �育ｴｯ遨咲ｵｱ險�DB菴ｿ逕ｨ縲√ラ繧ｭ繝･繝｡繝ｳ繝井ｸｭ縺ｮ鬆ｻ蠎ｦ菴ｿ逕ｨ縺ｫ繧ｻ繝�ヨ��
#
#my @noun_list = $data->get_imp_word($str, 'var'); # 蜈･蜉帙′螟画焚
my @noun_list = $data->get_imp_word($InputFile); # 蜈･蜉帙′繝輔ぃ繧､繝ｫ
#
# 蜑榊屓隱ｭ縺ｿ霎ｼ繧薙□蜩∬ｩ槭ち繧ｰ莉倥￠貂医∩繝�く繧ｹ繝医ヵ繧｡繧､繝ｫ繧貞�縺ｫ
# 繝｢繝ｼ繝峨ｒ螟峨∴縺ｦ縲∝ｰる摩逕ｨ隱槭Μ繧ｹ繝医ｒ驟榊�縺ｫ霑斐☆
#$data->use_stat->no_frq;
#my @noun_list2 = $data->get_imp_word();
# 縺ｾ縺溘√◎縺ｮ邨先棡繧貞挨縺ｮ繝｢繝ｼ繝峨↓繧医ｋ邨先棡縺ｨ謗帙￠蜷医ｏ縺帙ｋ
#@noun_list = $data->result_filter (\@noun_list, \@noun_list2, 30, 1000);
#
# 蟆る摩逕ｨ隱槭Μ繧ｹ繝医→險育ｮ励＠縺滄㍾隕∝ｺｦ繧呈ｨ呎ｺ門�蜉帙↓蜃ｺ縺�
#
foreach (@noun_list) {
# 謨ｰ蛟､縺ｮ縺ｿ縺ｯ陦ｨ遉ｺ縺励↑縺�
next if $_->[0] =~ /^\d+$/;
# 邨先棡陦ｨ遉ｺ
printf "%-60s %16.2f\n", $_->[0], $_->[1] if $output_mode == 1;
printf "%s\n", $_->[0] if $output_mode == 2;
printf "%s,", $_->[0] if $output_mode == 3;
}
縺薙�繝｢繧ｸ繝･繝ｼ繝ｫ縺ｧ縺ｯ縲“et_imp_word 縺ｮ縺ｿ螳溯｣�＠縲√◎繧御ｻ･螟悶�繝｡繧ｽ繝�ラ縺ｯ隕ｪ
繝｢繧ｸ繝･繝ｼ繝ｫ TermExtract::Calc_Imp 縺ｧ螳溯｣�＆繧後※縺�ｋ縲�
get_imp_word 縺ｯ蜩∬ｩ槭ち繧ｰ莉倅ｸ弱ｒ陦後＞謚ｽ蜃ｺ縺輔ｌ縺溷腰隱槭ｒ縲∝九��蜊倩ｪ槭�隱樣��
縺ｨ蜩∬ｩ樊ュ蝣ｱ繧貞�縺ｫ隍�粋隱槭↓逕滓�縺励※縺�ｋ縲ゅ◎繧御ｻ･螟悶�繝｡繧ｽ繝�ラ縺ｫ縺､縺�※縺ｯ縲�
TermExtract::Calc_Imp 縺ｮPOD繝峨く繝･繝｡繝ｳ繝医ｒ蜿ら�縺吶ｋ縺薙→縲�
闍ｱ譁��蜩∬ｩ槭ち繧ｰ莉倅ｸ守ｵ先棡繧呈ｬ｡縺ｮ繝ｫ繝ｼ繝ｫ縺ｫ繧医ｊ隍�粋隱槭↓逕滓�縺吶ｋ縲らｬｬ�大ｼ墓焚縺ｯ縲�
蜃ｦ逅�ｯｾ雎｡縺ｮ繝��繧ｿ縲∫ｬｬ�貞ｼ墓焚縺ｯ隨ｬ�大ｼ墓焚縺ｮ遞ｮ蛻･縺ｧ縺ゅｋ縲ゅョ繝輔か繝ｫ繝医〒縺ｯ縲∫ｬｬ��
蠑墓焚縺ｯ縲∝刀隧槭ち繧ｰ莉倥￠貂医∩縺ｮ繝�く繧ｹ繝医ヵ繧｡繧､繝ｫ縺ｨ縺ｪ繧九らｬｬ�貞ｼ墓焚縺ｫ譁�ｭ怜�
'var'縺後そ繝�ヨ縺輔ｌ縺溘→縺阪↓縺ｯ縲∫ｬｬ荳蠑墓焚繧貞刀隧槭ち繧ｰ莉倥￠貂医�繝�く繧ｹ繝医ョ繝ｼ繧ｿ
縺悟�縺｣縺溘せ繧ｫ繝ｩ繝ｼ螟画焚縺ｨ隗｣驥医☆繧九�
�托ｼ主推蜩∬ｩ槭�谺｡縺ｮ縺ｨ縺翫ｊ邨仙粋縺吶ｋ
�茨ｼ托ｼ牙錐隧�(NN) 縲縲縲竊偵蜷崎ｩ槭∝ｽ｢螳ｹ隧槭∝渕謨ｰ縲�℃蜴ｻ蛻�ｩ槭�蜍戊ｩ槭↓
邨仙粋縺吶ｋ縲り､�粋隱槭�蜈磯�ｭ縺ｫ縺ｪ繧九�
�茨ｼ抵ｼ牙､匁擂隱�(FW) 縲縲縲竊偵蜊倩ｪ槭→縺励※蜃ｦ逅�
�茨ｼ難ｼ牙渕謨ｰ(CD) 縲縲縲竊偵隍�粋隱槭�蜈磯�ｭ縺ｮ縺ｿ險ｱ蜿ｯ縺吶ｋ
�茨ｼ費ｼ牙ｽ｢螳ｹ隧�(JJ) 縲縲縲竊偵蠖｢螳ｹ隧�,謇譛画�ｼ隱槫ｰｾ,蝓ｺ謨ｰ縺ｫ邨仙粋縺吶ｋ縲�
隍�粋隱槭�蜈磯�ｭ縺ｫ縺ｪ繧�
(�包ｼ画園譛画�ｼ隱槫ｰｾ(POS)縲 縲竊偵蜷崎ｩ槭↓邨仙粋縺吶ｋ
�茨ｼ厄ｼ頴f縲縲縲縲縲縲縲縲縲竊偵蜷崎ｩ槭↓邨仙粋縺吶ｋ
�茨ｼ暦ｼ蛾℃蜴ｻ蛻�ｩ槭�蜍戊ｩ�(VBN) 竊偵隍�粋隱槭�蜈磯�ｭ縺ｮ縺ｿ險ｱ蜿ｯ縺吶ｋ
�抵ｼ取隼陦後′縺ゅ▲縺溷�ｴ蜷医�縲√◎縺薙〒隍�粋隱槭�蛹ｺ蛻�ｊ縺ｨ縺吶ｋ
�難ｼ取ｬ｡縺ｮ險伜捷繧�焚蛟､縺ｧ蟋九∪繧玖ｪ槭�蝣ｴ蜷医�縲√◎縺薙〒隍�粋隱槭�蛹ｺ蛻�ｊ縺ｨ縺吶ｋ
+-%\&\$*#^|
�費ｼ手､�粋隱槭�蜷崎ｩ槭°螟匁擂隱槭〒邨ゅｏ繧九ｂ縺ｮ縺ｨ縺励∽ｻ･蠕後�蛻�ｊ謐ｨ縺ｦ繧�
�包ｼ主崋譛牙錐隧樔ｻ･螟悶�蜷崎ｩ槭�縲∝�鬆ｭ縺悟､ｧ譁�ｭ励�蝣ｴ蜷医↓蟆乗枚蟄励↓螟画鋤縺吶ｋ
�厄ｼ手､�粋隱槭�蜷崎ｩ�(NNS)繧貞腰謨ｰ蠖｢縺ｫ螟峨∴繧�
�暦ｼ�' �医す繝ｳ繧ｰ繝ｫ繧ｯ繧ｩ繝ｼ繝��繧ｷ繝ｧ繝ｳ)縺ｧ蛹ｺ蛻�ｉ繧後◆隱槭�蜊倩ｪ槭→縺吶ｋ
�假ｼ手､�粋隱樊忰蟆ｾ縺ｮ , . 縺ｯ髯､蜴ｻ縺吶ｋ
�呻ｼ朱㍾隕∝ｺｦ險育ｮ励↓縺翫＞縺ｦ谺｡縺ｮ隱槭�辟｡隕悶☆繧�
of Of OF
TermExtract::Calc_Imp
TermExtract::Chasen
TermExtract::MeCab
TermExtract::EnglishPlainText
TermExtract::ChainesPlainTextUC
TermExtract::ChainesPlainTextGB
TermExtract::ICTCLAS
TermExtract::JapanesePlainTextEUC
TermExtract::JapanesePlainTextSJIS
縺薙�繝励Ο繧ｰ繝ｩ繝�縺ｯ縲∵擲莠ｬ螟ｧ蟄ｦ繝ｻ荳ｭ蟾晁｣募ｿ玲蕗謗医∵ｨｪ豬懷嵜遶句､ｧ蟄ｦ繝ｻ譽ｮ霎ｰ蜑�勧謨呎肢縺�
菴懈�縺励◆縲悟ｰる摩逕ｨ隱櫁�蜍墓歓蜃ｺ繧ｷ繧ｹ繝�Β縲阪�termex_e.pl 繧貞�縺ｫ繝｢繧ｸ繝･繝ｼ繝ｫ
TermExtract逕ｨ縺ｫ譖ｸ縺肴鋤縺医◆繧ゅ�縺ｧ縺ゅｋ縲�
縺薙�菴懈･ｭ縺ｯ縲∵擲莠ｬ螟ｧ蟄ｦ繝ｻ蜑咲伐譛� (maeda@lib.u-tokyo.ac.jp)縺瑚｡後▲縺溘�
縺ｪ縺翫∵悽繝励Ο繧ｰ繝ｩ繝�縺ｮ菴ｿ逕ｨ縺ｫ縺翫＞縺ｦ逕溘§縺溘＞縺九↑繧狗ｵ先棡縺ｫ髢｢縺励※繧ょｽ捺婿縺ｧ縺ｯ
荳蛻�ｲｬ莉ｻ繧定ｲ�繧上↑縺��