Performance considerations

A short summary of what makes Datashare fast or slow, and where the detailed guidance lives.

Extracting text from arbitrary file formats and running OCR over images is expensive work. Datashare uses Apache Tika and Tesseract OCR to do it as efficiently as it can, but on a large corpus the settings you choose matter more than the machine you rent.

The detailed guidance now lives in Tuning and performance. In short:

• Match threads to cores. Set --parallelism to the number of cores and make sure OMP_THREAD_LIMIT=1 reaches Tesseract. Getting this wrong is routinely a 30x slowdown. Read more.

• Decide about OCR. It is on by default and it dominates the cost. Turn it off for born-digital corpora, and install the right Tesseract language packs when you keep it on. Read more.

• Split the stages and distribute INDEX. With a shared Redis queue and a shared Elasticsearch, adding machines adds throughput almost linearly. Read more.

• Give Elasticsearch its own machine. It becomes the bottleneck before your indexing workers do. Read more.

• Size the heap deliberately with DS_JAVA_OPTS and ES_JAVA_OPTS. Read more.

• Treat mail archives as a special case. One mailbox file is one queue entry, so a corpus of a few very large mailboxes leaves a big machine underused. Read more.

• Tell Datashare the language when your corpus is monolingual, with --language and --ocrLanguage. Read more.