Estraier(3pm)
NAME
Estraier - Perl Binding of Hyper Estraier
SYNOPSYS
use Estraier;
INTRODUCTION
Hyper Estraier is a full-text search system for communities.
This is a package implementing the core API of Hyper Estraier (
http://hyperestraier.sourceforge.net/ ), including native codes written
in C with XS macros. As it works on Linux, Mac OS X, Windows, and so
on, native libraries for each environment are required to run programs.
This package requires Perl 5.8.8 or later versions.
Setting
Install the latest version of Hyper Estraier.
Enter the sub directory `perlnative' in the extracted package then
perform installation.
cd perlnative
./configure
make
su
make install
On Linux and other UNIX systems: set the environment variable
LD_LIBRARY_PATH to find libraries; "libestraier.so". On Mac OS X: set
the environment variable DYLD_LIBRARY_PATH to find libraries;
"libestraier.dylib". On Windows: set the environment variable PATH to
find libraries; "estraier.dll".
The package `Estraier' should be loaded in each source file of
application programs.
use Estraier;
If you want to enable runtime assertion, set the variable
`$Estraier::DEBUG' to be true.
$Estraier::DEBUG = 1;
DESCRIPTION
Class Document
$doc = new Document(draft)
Create a document object. `draft' specifies a string of draft
data. If it is omitted, an empty document object is created.
$doc->add_attr(name, value)
Add an attribute. `name' specifies the name of an attribute.
`value' specifies the value of the attribute. If it is `undef',
the attribute is removed. The return value is always `undef'.
$doc->add_text(text)
Add a sentence of text. `text' specifies a sentence of text. The
return value is always `undef'.
$doc->add_hidden_text(text)
Add a hidden sentence. `text' specifies a hidden sentence. The
return value is always `undef'.
$doc->set_keywords(kwords)
Attach keywords. `kwords' specifies the reference of a hash object
of keywords. Keys of the hash should be keywords of the document
and values should be their scores in decimal string. The return
value is always `undef'.
$doc->set_score(score)
Set the substitute score. `score' specifies the substitute score.
It should be zero or positive. The return value is always `undef'.
$doc->id()
Get the ID number. The return value is the ID number of the
document object. If the object has never been registered, -1 is
returned.
$doc->attr_names()
Get an array of attribute names of a document object. The return
value is a reference of an array object of attribute names.
$doc->attr()
Get the value of an attribute. `name' specifies the name of an
attribute. The return value is the value of the attribute or
`undef' if it does not exist.
$doc->texts()
Get an array of sentences of the text. The return value is a
reference of an array object of sentences of the text.
$doc->cat_texts()
Concatenate sentences of the text of a document object. The return
value is concatenated sentences.
$doc->keywords()
Get attached keywords. The return value is a reference of a hash
object of keywords and their scores in decimal string. If no
keyword is attached, `undef' is returned.
$doc->score()
Get the substitute score. The return value is the substitute score
or -1 if it is not set.
$doc->dump_draft()
Dump draft data of a document object. The return value is draft
data.
$doc->make_snippet(words, wwidth, hwidth, awidth)
Make a snippet of the body text. `words' specifies a reference of
an array object of words to be highlight. `wwidth' specifies whole
width of the result. `hwidth' specifies width of strings picked up
from the beginning of the text. `awidth' width of strings picked
up around each highlighted word. The return value is a snippet
string of the body text. There are tab separated values. Each
line is a string to be shown. Though most lines have only one
field, some lines have two fields. If the second field exists, the
first field is to be shown with highlighted, and the second field
means its normalized form.
Class Condition
Condition::SURE = 1 << 0
option: check every N-gram key
Condition::USUAL = 1 << 1
option: check N-gram keys skipping by one
Condition::FAST = 1 << 2
option: check N-gram keys skipping by two
Condition::AGITO = 1 << 3
option: check N-gram keys skipping by three
Condition::NOIDF = 1 << 4
option: without TF-IDF tuning
Condition::SIMPLE = 1 << 10
option: with the simplified phrase
Condition::ROUGH = 1 << 11
option: with the rough phrase
Condition::UNION = 1 << 15
option: with the union phrase
Condition::ISECT = 1 << 16
option: with the intersection phrase
Condition::ECLSIMURL = 10.0
eclipse tuning: consider URL
Condition::ECLSERV = 100.0
eclipse tuning: on server basis
Condition::ECLDIR = 101.0
eclipse tuning: on directory basis
Condition::ECLFILE = 102.0
eclipse tuning: on file basis
$cond = new Condition()
Create a search condition object.
$cond->set_phrase(phrase)
Set the search phrase. `phrase' specifies a search phrase. The
return value is always `undef'.
$cond->add_attr(expr)
Add an expression for an attribute. `expr' specifies an expression
for an attribute. The return value is always `undef'.
$cond->set_order(expr)
Set the order of a condition object. `expr' specifies an
expression for the order. By default, the order is by score
descending. The return value is always `undef'.
$cond->set_max(max)
Set the maximum number of retrieval. `max' specifies the maximum
number of retrieval. By default, the number of retrieval is not
limited.
$cond->set_skip(skip)
Set the number of skipped documents. `skip' specifies the number
of documents to be skipped in the search result. The return value
is always `undef'.
$cond->set_options(options)
Set options of retrieval. `options' specifies options:
`Condition::SURE' specifies that it checks every N-gram key,
`Condition::USU', which is the default, specifies that it checks
N-gram keys with skipping one key, `Condition::FAST' skips two
keys, `Condition::AGITO' skips three keys, `Condition::NOIDF'
specifies not to perform TF-IDF tuning, `Condition::SIMPLE'
specifies to use simplified phrase, `Condition::ROUGH' specifies to
use rough phrase, `Condition::UNION' specifies to use union phrase,
`Condition::ISECT' specifies to use intersection phrase. Each
option can be specified at the same time by bitwise or. If keys
are skipped, though search speed is improved, the relevance ratio
grows less. The return value is always `undef'.
$cond->set_auxiliary(min)
Set permission to adopt result of the auxiliary index. `min'
specifies the minimum hits to adopt result of the auxiliary index.
If it is not more than 0, the auxiliary index is not used. By
default, it is 32.
$cond->set_eclipse(limit)
Set the lower limit of similarity eclipse. `limit' specifies the
lower limit of similarity for documents to be eclipsed. Similarity
is between 0.0 and 1.0. If the limit is added by
`Condition::ECLSIMURL', similarity is weighted by URL. If the
limit is `Condition::ECLSERV', similarity is ignored and documents
in the same server are eclipsed. If the limit is
`Condition::ECLDIR', similarity is ignored and documents in the
same directory are eclipsed. If the limit is `Condition::ECLFILE',
similarity is ignored and documents of the same file are eclipsed.
$cond->set_distinct(name)
Set the attribute distinction filter. `name' specifies the name of
an attribute to be distinct. The return value is always `undef'.
Class Result
$result->doc_num()
Get the number of documents. The return value is the number of
documents in the result.
$result->get_doc_id(index)
Get the ID number of a document. `index' specifies the index of a
document. The return value is the ID number of the document or -1
if the index is out of bounds.
$result->get_dbidx(index)
Get the index of the container database of a document. `index'
specifies the index of a document. The return value is the index
of the container database of the document or -1 if the index is out
of bounds.
$result->hint_words()
Get an array of hint words. The return value is a reference of an
array of hint words.
$result->hint(word)
Get the value of a hint word. `word' specifies a hint word. An
empty string means the number of whole result. The return value is
the number of documents corresponding the hint word. If the word
is in a negative condition, the value is negative.
$result->get_score(index)
Get the score of a document. `index' specifies the index of a
document. The return value is the score of the document or -1 if
the index is out of bounds.
$result->get_shadows(id)
Get an array of ID numbers of eclipsed docuemnts of a document.
`id' specifies the ID number of a parent document. The return
value is a reference of an array whose elements expresse the ID
numbers and their scores alternately.
Class Database
Database::VERSION = "0.0.0"
version of Hyper Estraier
Database::ERRNOERR = 0
error code: no error
Database::ERRINVAL = 1
error code: invalid argument
Database::ERRACCES = 2
error code: access forbidden
Database::ERRLOCK = 3
error code: lock failure
Database::ERRDB = 4
error code: database problem
Database::ERRIO = 5
error code: I/O problem
Database::ERRNOITEM = 6
error code: no item
Database::ERRMISC = 9999
error code: miscellaneous
Database::DBREADER = 1 << 0
open mode: open as a reader
Database::DBWRITER = 1 << 1
open mode: open as a writer
Database::DBCREAT = 1 << 2
open mode: a writer creating
Database::DBTRUNC = 1 << 3
open mode: a writer truncating
Database::DBNOLCK = 1 << 4
open mode: open without locking
Database::DBLCKNB = 1 << 5
open mode: lock without blocking
Database::DBPERFNG = 1 << 10
open mode: use perfect N-gram analyzer
Database::DBCHRCAT = 1 << 11
open mode: use character category analyzer
Database::DBSMALL= 1 << 20
open mode: small tuning
Database::DBLARGE = 1 << 21
open mode: large tuning
Database::DBHUGE = 1 << 22
open mode: huge tuning
Database::DBHUGE2 = 1 << 23
open mode: huge tuning second
Database::DBHUGE3 = 1 << 24
open mode: huge tuning third
Database::DBSCVOID = 1 << 25
open mode: store scores as void
Database::DBSCINT = 1 << 26
open mode: store scores as integer
Database::DBSCASIS = 1 << 27
open mode: refrain from adjustment of scores
Database::IDXATTRSEQ = 0
attribute index type: for multipurpose sequential access method
Database::IDXATTRSTR = 1
attribute index type: for narrowing with attributes as strings
Database::IDXATTRNUM = 2
attribute index type: for narrowing with attributes as numbers
Database::OPTNOPURGE = 1 << 0
optimize option: omit purging dispensable region of deleted
Database::OPTNODBOPT = 1 << 1
optimize option: omit optimization of the database files
Database::MGCLEAN = 1 << 0
merge option: clean up dispensable regions
Database::PDCLEAN = 1 << 0
put_doc option: clean up dispensable regions
Database::PDWEIGHT = 1 << 1
put_doc option: weight scores statically when indexing
Database::ODCLEAN = 1 << 0
out_doc option: clean up dispensable regions
Database::GDNOATTR = 1 << 0
get_doc option: no attributes
Database::GDNOTEXT = 1 << 1
get_doc option: no text
Database::GDNOKWD = 1 << 2
get_doc option: no keywords
$db = new Database()
Create a database object.
Database::search_meta(dbs, cond)
Search plural databases for documents corresponding a condition.
`dbs' specifies a reference of an array whose elements are database
objects. `cond' specifies a condition object. The return value is
a result object. On error, `undef' is returned.
$db->err_msg(ecode)
Get the string of an error code. `ecode' specifies an error code.
The return value is the string of the error code.
$db->open(name, omode)
Open a database. `name' specifies the name of a database
directory. `omode' specifies open modes: `Database::DBWRITER' as a
writer, `Database::DBREADER' as a reader. If the mode is
`Database::DBWRITER', the following may be added by bitwise or:
`Database::DBCREAT', which means it creates a new database if not
exist, `Database::DBTRUNC', which means it creates a new database
regardless if one exists. Both of `Database::DBREADER' and
`Database::DBWRITER' can be added to by bitwise or:
`Database::DBNOLCK', which means it opens a database file without
file locking, or `Database::DBLCKNB', which means locking is
performed without blocking. If `Database::DBNOLCK' is used, the
application is responsible for exclusion control.
`Database::DBCREAT' can be added to by bitwise or:
`Database::DBPERFNG', which means N-gram analysis is performed
against European text also, `Database::DBCHACAT', which means
character category analysis is performed instead of N-gram
analysis, `Database::DBSMALL', which means the index is tuned to
register less than 50000 documents, `Database::DBLARGE', which
means the index is tuned to register more than 300000 documents,
`Database::DBHUGE', which means the index is tuned to register more
than 1000000 documents, `Database::DBHUGE2', which means the index
is tuned to register more than 5000000 documents,
`Database::DBHUGE3', which means the index is tuned to register
more than 10000000 documents, `Database::DBSCVOID', which means
scores are stored as void, `Database::DBSCINT', which means scores
are stored as 32-bit integer, `Database::DBSCASIS', which means
scores are stored as-is and marked not to be tuned when search.
The return value is true if success, else it is false.
$db->close()
Close the database. The return value is true if success, else it
is false.
$db->error()
Get the last happened error code. The return value is the last
happened error code.
$db->fatal()
Check whether the database has a fatal error. The return value is
true if the database has fatal erroor, else it is false.
$db->add_attr_index(name, type)
Add an index for narrowing or sorting with document attributes.
`name' specifies the name of an attribute. `type' specifies the
data type of attribute index; `Database::IDXATTRSEQ' for
multipurpose sequential access method, `Database::IDXATTRSTR' for
narrowing with attributes as strings, `Database::IDXATTRNUM' for
narrowing with attributes as numbers. The return value is true if
success, else it is false.
$db->flush(max)
Flush index words in the cache. `max' specifies the maximum number
of words to be flushed. If it not more than zero, all words are
flushed. The return value is true if success, else it is false.
$db->sync()
Synchronize updating contents. The return value is true if
success, else it is false.
$db->optimize(options)
Optimize the database. `options' specifies options:
`Database::OPTNOPURGE' to omit purging dispensable region of
deleted documents, `Database::OPTNODBOPT' to omit optimization of
the database files. The two can be specified at the same time by
bitwise or. The return value is true if success, else it is false.
$db->merge(name, options)
Merge another database. `name' specifies the name of another
database directory. `options' specifies options:
`Database::MGCLEAN' to clean up dispensable regions of the deleted
document. The return value is true if success, else it is false.
$db->put_doc(doc, options)
Add a document. `doc' specifies a document object. The document
object should have the URI attribute. `options' specifies options:
`Database::PDCLEAN' to clean up dispensable regions of the
overwritten document. The return value is true if success, else it
is false.
$db->out_doc(id, options)
Remove a document. `id' specifies the ID number of a registered
document. `options' specifies options: `Database::ODCLEAN' to
clean up dispensable regions of the deleted document. The return
value is true if success, else it is false.
$db->edit_doc(doc)
Edit attributes of a document. `doc' specifies a document object.
The return value is true if success, else it is false.
$db->get_doc(id, options)
Retrieve a document. `id' specifies the ID number of a registered
document. `options' specifies options: `Database::GDNOATTR' to
ignore attributes, `Database::GDNOTEXT' to ignore the body text,
`Database::GDNOKWD' to ignore keywords. The three can be specified
at the same time by bitwise or. The return value is a document
object. On error, `undef' is returned.
$db->get_doc_attr(id, name)
Retrieve the value of an attribute of a document. `id' specifies
the ID number of a registered document. `name' specifies the name
of an attribute. The return value is the value of the attribute or
`undef' if it does not exist.
$db->uri_to_id(uri)
Get the ID of a document specified by URI. `uri' specifies the URI
of a registered document. The return value is the ID of the
document. On error, -1 is returned.
$db->name()
Get the name. The return value is the name of the database.
$db->doc_num()
Get the number of documents. The return value is the number of
documents in the database.
$db->word_num()
Get the number of unique words. The return value is the number of
unique words in the database.
$db->size()
Get the size. The return value is the size of the database.
$db->search(cond)
Search for documents corresponding a condition. `cond' specifies a
condition object. The return value is a result object. On error,
`undef' is returned.
$db->scan_doc(doc, cond)
Check whether a document object matches the phrase of a search
condition object definitely. `doc' specifies a document object.
`cond' specifies a search condition object. The return value is
true if the document matches the phrase of the condition object
definitely, else it is false.
$db->set_cache_size(size, anum, tnum, rnum)
Set the maximum size of the cache memory. `size' specifies the
maximum size of the index cache. By default, it is 64MB. If it is
not more than 0, the current size is not changed. `anum' specifies
the maximum number of cached records for document attributes. By
default, it is 8192. If it is not more than 0, the current size is
not changed. `tnum' specifies the maximum number of cached records
for document texts. By default, it is 1024. If it is not more
than 0, the current size is not changed. `rnum' specifies the
maximum number of cached records for occurrence results. By
default, it is 256. If it is not more than 0, the current size is
not changed. The return value is always `undef'.
$db->add_pseudo_index(path)
Add a pseudo index directory. `path' specifies the path of a
pseudo index directory. The return value is true if success, else
it is false.
$db->set_wildmax(num)
Set the maximum number of expansion of wild cards. `num' specifies
the maximum number of expansion of wild cards. The return value is
always `undef'.
$db->set_informer(informer)
Set the callback function to inform of database events. `informer'
specifies the name of an arbitrary function. The function should
have one parameter for a string of a message of each event. The
return value is always `undef'.
EXAMPLE
Gatherer
The following is the simplest implementation of a gatherer.
use strict;
use warnings;
use Estraier;
$Estraier::DEBUG = 1;
# create the database object
my $db = new Database();
# open the database
unless($db->open("casket", Database::DBWRITER | Database::DBCREAT)){
printf("error: %s\n", $db->err_msg($db->error()));
exit;
}
# create a document object
my $doc = new Document();
# add attributes to the document object
$doc->add_attr('@uri', "https://estraier.gov/example.txt");
$doc->add_attr('@title', "Over the Rainbow");
# add the body text to the document object
$doc->add_text("Somewhere over the rainbow. Way up high.");
$doc->add_text("There's a land that I heard of once in a lullaby.");
# register the document object to the database
unless($db->put_doc($doc, Database::PDCLEAN)){
printf("error: %s\n", $db->err_msg($db->error()));
}
# close the database
unless($db->close()){
printf("error: %s\n", $db->err_msg($db->error()));
}
Searcher
The following is the simplest implementation of a searcher.
use strict;
use warnings;
use Estraier;
$Estraier::DEBUG = 1;
# create the database object
my $db = new Database();
# open the database
unless($db->open("casket", Database::DBREADER)){
printf("error: %s\n", $db->err_msg($db->error()));
exit;
}
# create a search condition object
my $cond = new Condition();
# set the search phrase to the search condition object
$cond->set_phrase("rainbow AND lullaby");
# get the result of search
my $result = $db->search($cond);
# for each document in the result
my $dnum = $result->doc_num();
foreach my $i (0..$dnum-1){
# retrieve the document object
my $doc = $db->get_doc($result->get_doc_id($i), 0);
next unless(defined($doc));
# display attributes
my $uri = $doc->attr('@uri');
printf("URI: %s\n", $uri) if defined($uri);
my $title = $doc->attr('@title');
printf("Title: %s\n", $title) if defined($title);
# display the body text
my $texts = $doc->texts();
foreach my $text (@$texts){
printf("%s\n", $text);
}
}
# close the database
unless($db.close()){
printf("error: %s\n", $db->err_msg($db->error()));
}
LICENSE
Copyright (C) 2004-2007 Mikio Hirabayashi
All rights reserved.
Hyper Estraier is free software; you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License as
published by the Free Software Foundation; either version 2.1 of the
License or any later version. Hyper Estraier is distributed in the
hope that it will be useful, but WITHOUT ANY WARRANTY; without even the
implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE. See the GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with Hyper Estraier; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
USA.
Opportunity
Personal Opportunity - Free software gives you access to billions of dollars of software at no cost. Use this software for your business, personal use or to develop a profitable skill. Access to source code provides access to a level of capabilities/information that companies protect though copyrights. Open source is a core component of the Internet and it is available to you. Leverage the billions of dollars in resources and capabilities to build a career, establish a business or change the world. The potential is endless for those who understand the opportunity.
Business Opportunity - Goldman Sachs, IBM and countless large corporations are leveraging open source to reduce costs, develop products and increase their bottom lines. Learn what these companies know about open source and how open source can give you the advantage.
Free Software
Free Software provides computer programs and capabilities at no cost but more importantly, it provides the freedom to run, edit, contribute to, and share the software. The importance of free software is a matter of access, not price. Software at no cost is a benefit but ownership rights to the software and source code is far more significant.
Free Office Software - The Libre Office suite provides top desktop productivity tools for free. This includes, a word processor, spreadsheet, presentation engine, drawing and flowcharting, database and math applications. Libre Office is available for Linux or Windows.
Free Books
The Free Books Library is a collection of thousands of the most popular public domain books in an online readable format. The collection includes great classical literature and more recent works where the U.S. copyright has expired. These books are yours to read and use without restrictions.
Source Code - Want to change a program or know how it works? Open Source provides the source code for its programs so that anyone can use, modify or learn how to write those programs themselves. Visit the GNU source code repositories to download the source.
Education
Study at Harvard, Stanford or MIT - Open edX provides free online courses from Harvard, MIT, Columbia, UC Berkeley and other top Universities. Hundreds of courses for almost all major subjects and course levels. Open edx also offers some paid courses and selected certifications.
Linux Manual Pages - A man or manual page is a form of software documentation found on Linux/Unix operating systems. Topics covered include computer programs (including library and system calls), formal standards and conventions, and even abstract concepts.
