Need some information about Ferret

[Lyes Amazouz]

Hi everybody!

In our company, we want to use Ferret as the main index/search engine of our
applications. And we are looking for some testimonies about how Ferret is
efficient when deployed in production.

* Was Ferret already deployed in production in some companies? is there some
testimonies about that?

* What is the maximum number of documents we can index with ferret? Has some
one informations about that.

* What is the best way to access a very huge Ferret Index? May we distribute
it on several machines or not?

By the way, can Ferret read Solr indexes as they are both clones of luceen?

thank you

[Erik Hatcher]

Post by Lyes Amazouz
By the way, can Ferret read Solr indexes as they are both clones of luceen?

No. While Ferret was designed around the Lucene index file format, it
is not compatible with Java Lucene (and thus Solr).

Erik

[Lyes Amazouz]

On Sun, Nov 30, 2008 at 10:48 AM, Erik Hatcher

Post by Lyes Amazouz
By the way, can Ferret read Solr indexes as they are both clones of luceen?

No. While Ferret was designed around the Lucene index file format, it is
not compatible with Java Lucene (and thus Solr).
Erik
_______________________________________________
Ferret-talk mailing list
http://rubyforge.org/mailman/listinfo/ferret-talk

Hello Eik!

thank you for the information. But is there a mean to recover an existing
Solr index content and reindex it with Ferret?

[Erik Hatcher]

Post by Lyes Amazouz
thank you for the information. But is there a mean to recover an
existing Solr index content and reindex it with Ferret?

It'll probably be easier and faster to reindex your original content,
which presumably you still have handy. But... you'd have to have your
fields "stored" in Solr for them to be recoverable. Using solr-ruby's
Solr::Importer::SolrSource would makes it easy to iterate over all
documents in Solr (using a query of *:*).

But why move from Solr to Ferret?

Erik

[Lyes Amazouz]

Hello

But why move from Solr to Ferret?
We found that the search and the indexation with Solr was too slow, and we
decided to find another alternative. Ferret seems to be a good choice. We
tried Ferret on some examples and we found that it was better.

[Erik Hatcher]

Post by Erik Hatcher
Hello
But why move from Solr to Ferret?
We found that the search and the indexation with Solr was too slow,
and we decided to find another alternative. Ferret seems to be a
good choice. We tried Ferret on some examples and we found that it
was better.

Thanks for the feedback. If you don't mind elaborating further, what
kind of documents are you indexing (database rows? file system
files? other?), how many documents do you have, and how are you
indexing it?

Thanks,
Erik

[Lyes Amazouz]

Hello Erik

Thanks for the feedback. If you don't mind elaborating further, what kind

of documents are you indexing (database rows? file system files? other?),
how many documents do you have, and how are you indexing it?
Thanks,
Erik

Now, we are indexing file system files varying from HTML pages (85%) to
IMAGES (10%) (We index Meta information here), PDF(2%) WORD (2%) and PURE
TEXT (1%), we have 100 000 000 documents to index (10%) is already done. And
for the last question, I didn't exactly understand what do you mean by "How
we are indexing", What I can say is that before we index non full text
documents (like PDF, WORD and HTML), we operate a content extraction
(usingpdftotext, antiword and 'hpricot' ruby library). We axtract also the
metadata related to each document we index.

_______________________________________________
Ferret-talk mailing list
http://rubyforge.org/mailman/listinfo/ferret-talk

[Jens Krämer]

Hi!

Post by Lyes Amazouz
Hi everybody!
In our company, we want to use Ferret as the main index/search
engine of our applications. And we are looking for some testimonies
about how Ferret is efficient when deployed in production.
* Was Ferret already deployed in production in some companies? is
there some testimonies about that?

Yes, I use Ferret whenever I need some kind of search for a site or
application I'm working on. Usually these are full text searches for
product catalogs and/or html content - not really large scale, at most
around 10000 documents. Most recent example is www.fahrrad-xxl.de.

We also use Ferret + aaf in a knowledge management system I'm working
on for xscio AG (xscio.de).

Post by Lyes Amazouz
* What is the maximum number of documents we can index with ferret?
Has some one informations about that.

I have no idea whether there is an upper limit for the number the
documents other than the maximum value a Ruby Fixnum instance can
have...

Post by Lyes Amazouz
* What is the best way to access a very huge Ferret Index? May we
distribute it on several machines or not?

Afair there's no way to distribute an index across multiple machines
built into Ferret. You could do the distribution yourself of course by
clustering your data and distributing across several independent
ferret indexes. Downside is that search result scores from different
indexes aren't directly comparable.

Post by Lyes Amazouz
By the way, can Ferret read Solr indexes as they are both clones of luceen?

Ferret isn't really index compatible with Lucene anymore, it uses a
slightly different index format mostly due to differences in the
representation of utf8 values, but I think there were other changes,
too.

Oh, and Solr also isn't a clone of Lucene, it's a search server that
internally uses the Lucene library.


Cheers,
Jens

--
Jens Krämer
webit! Gesellschaft für neue Medien mbH
Schnorrstraße 76 | 01069 Dresden
Telefon [phone] | Telefax [phone]
kraemer-jv+***@public.gmane.org | www.webit.de

Amtsgericht Dresden | HRB 15422
GF Sven Haubold

[Lyes Amazouz]

Hello Jens!

Thank you for your contribution.

Post by Jens Krämer
Yes, I use Ferret whenever I need some kind of search for a site or
application I'm working on. Usually these are full text searches for product
catalogs and/or html content - not really large scale, at most around 10000
documents. Most recent example is www.fahrrad-xxl.de.

Is 100 000 your maximum documents Number?

We have more than 100.000.000 documents to index. 2.800.000 are already
done but the indexation machine starts to be heavy! Do you think that ferret
will be able to index all this?

Post by Jens Krämer
* What is the best way to access a very huge Ferret Index? May we

Post by Lyes Amazouz
distribute it on several machines or not?

Afair there's no way to distribute an index across multiple machines built
into Ferret. You could do the distribution yourself of course by clustering
your data and distributing across several independent ferret indexes.
Downside is that search result scores from different indexes aren't directly
comparable.

Yes, it is a good Idea. But how will we merge the results when we will get
them back after a request?