Add entities from the CLI

This document assumes that you have installed Datashare in server mode within Docker and already added documents to Datashare.

In server mode, it's important to understand that Datashare does not provide a web interface to add documents or extract entities. Named entities are extracted using the command-line interface.

Datashare has the ability to detect email addresses, name of people, organizations and locations. This process use a Natural Language Processing (NLP) pipeline called CORENLP. Once your documents have been indexed in Datashare, you can perform the named entities extraction in the same fashion as the previous CLI's stages:

docker compose exec datashare_web /entrypoint.sh \
  stage run \
  --stages NLP \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --nlpParallelism 2 \
  --nlpPipeline CORENLP

What's happening here:

• We ask to process the NLP stage

• We tell Datashare to use the elasticsearch service

• Datashare will pull documents from ElasticSearch directly

• Up to 2 documents will be analyzed in parallel

• Datashare will use the CORENLP pipeline

Datashare will use the output queue from the previous INDEX stage (by default extract:queue:nlp in Redis) that contains all the document ids to be analyzed.

The first time you run this command you will have to wait a little bit because Datashare need to download CORENLP's models which can be big.

You can also use chain the 3 stages altogether:

docker compose exec datashare_web /entrypoint.sh \
  stage run \
  --stages SCAN,INDEX,NLP \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --nlpParallelism 2 \
  --nlpPipeline CORENLP \
  --dataDir /home/datashare/Datashare/

As for the previous stages you may want to restore the output queue from the INDEX stage. You can do:

docker compose exec datashare_web /entrypoint.sh \
  stage run \
  --stages ENQUEUEIDX,NLP \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --nlpParallelism 2 \
  --nlpPipeline CORENLP

The added ENQUEUEIDX stage will read Elasticsearch index, find all documents that have not already been analyzed by the CORENLP NER pipeline, and put the IDs of those documents into the extract:queue:nlp queue.