This page list all the concepts implemented by Datashare that users might want to understand before starting to search within documents.
When running Datashare from the command-line, pick which "stage" to apply to analyse your documents.
The CLI stages are primarily intented to be run for an instance of Datashare that uses non-embedded resources (ElasticSearch, database, key/value memory store). This allows you to distribute heavy tasks between servers.
This is the first step to add documents to Datashare from the command-line. The SCAN stage allows you to queue all the files that need to be indexed (next step). Once this task is done, you can move to the next step. This stage cannot be distributed.
datashare --mode CLI \
# Select the SCAN stage
--stage SCAN \
# Where the document are located
--dataDir /path/to/documents \
# Store the queued files in Redis
--dataBusType REDIS \
# URI of Redis
--redisAddress redis://redis:6379The INDEX stage is probably the most important (and heavy!) one. It pulls documents to index from the queue created in the previous step, then use a combination of and to extract text, metadata and OCR images. The result documents are stored in ElasticSearch. The queue used to store documents to index is a "blocking list", meaning that only one client can pull a concurrent value at the time. This allows users to distribute this command on several servers.
Once a document is available for search (stored in ElasticSearch), you can use the NLP stage to extract named entities from the text. This process will not only create named entity mentions in ElasticSearch, it will also mark every analyzed document with the corresponding NLP pipeline (CORENLP by default). In other words, the process is idempotent and can be parallelized as well on several servers.
Datashare runs using different modes with their own features.
LOCAL
Web
To run Datashare on a single computer for a single user.
SERVER
Web
To run Datashare on a server for multiple users.
CLI
CLI
To index documents and analyze them directly .
There are two modes:
In local mode and embedded mode, Datashare provides a self-contained software application that users can install and run on their own local machines. The software allows users to search into their documents within their own local environments, without relying on external servers or cloud infrastructure. This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.
In server mode, Datashare operates as a centralized server-based system. Users can access to the platform through a web interface, and the documents are stored and processed on Datashare's servers. This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.
The running modes offer advantages and limitations. This matrix summarizes the differences:
When running Datashare in local mode, users can choose to use embedded services (like ElasticSearch, SQLITE, in-memory key/value store) on the same JVM than Datashare. This variant of the local mode is called "embedded mode" and allows user to run Datashare without having to setup any additional software. The embedded mode is used by default.
In cli mode, Datashare starts without a web server and allows user to perform tasks over their documents. This mode can be used in conjunction with both local and server modes, while allowing users to distribute heavy tasks between several servers.
If you want to learn more about which tasks you can execute in this mode, checkout the .
Those modes are intended to be used for action that requires to "wait" for pendings tasks.
In batch download mode, the daemon waits for a user to request a batch download of documents. When a request is received, the daemon starts a task to download the document matching the user search, and bundle them into a zip file.
In batch search mode, the daemon waits for a user to request a batch search of documents. To create a batch search, users must go through the dedicated form on Datashare where they can upload a list of search terms (in CSV format). The daemon will then start a task to search all matching documents and store every occurrences in the database.
Datashare is shipped as a single executable, with all modes available. As previously mentioned, the default mode is the embedded mode. Yet when starting Datashare in command line, you can explicitly specify the running mode. For instance on Ubuntu/Debian:
datashare --mode CLI \
# Select the INDEX stage
--stage INDEX \
# Where the document are located
--dataDir /path/to/documents \
# Store the queued files in Redis
--dataBusType REDIS \
# URI of Elasticsearch
--elasticsearchAddress http://elasticsearch:9200 \
# Enable OCR \
--ocr true
# URI of Redis
--redisAddress redis://redis:6379✅
❌
Extension UI
✅
❌
HTTP API
✅
✅
API Key
✅
✅
Single JVM
✅
❌
Tasks execution
✅
❌
TASK_RUNNER
Daemon
To execute async tasks (batch searches, batch downloads, scan, index, NER extraction, ...)
local
server
Multi-users
❌
✅
Multi-projects
✅
✅
Access-control
❌
✅
Indexing UI
✅
❌
Plugins UI
datashare --mode CLI \
# Select the NLP stage
--stage NLP \
# Use CORENLP to detect named entities
--nlpp CORENLP \
# URI of Elasticsearch
--elasticsearchAddress http://elasticsearch:9200 datashare \
# Switch to SERVER mode
--mode SERVER \
# Dummy session filter to creates ephemeral users
--authFilter org.icij.datashare.session.YesCookieAuthFilter \
# Name of the default project for every user
--defaultProject local-datashare \
# URI of Elasticsearch
--elasticsearchAddress http://elasticsearch:9200 \
# URI of Redis
--redisAddress redis://redis:6379 \
# store user sessions in Redis.
--sessionStoreType REDIS