How do I index very large file?

[femto Zheng]

Hello all, I'm doing a monitor application,
which fetches log file of application and indexing it,
how do I index very large file?,
like up to serveral GB. because the application logs
may log very large file in short time.