In local mode, Datashare provides a self-contained software application that users can install and run on their own local machines. The software allows users to search into their documents within their own local environments, without relying on external servers or cloud infrastructure. This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.

• your Operating System (Mac, Windows or Linux)

• the version of your Operating System

• the version of Datashare

• screenshots of your issue

• a description of your issue.

The installer will setup:

• MacPorts (if neither Homebrew or MacPorts are installed)

• Apache Tesseract with MacPorts or Homenrew

• Java JRE 17

• Datashare executable

1. Download Datashare

2. Start the installer

Go to your "Downloads" directory in Finder and double-click "datashare-X.Y.Z.pkg":

3. Go through the Datashare Installer

Click 'Continue', 'Install', enter your password and 'Install Software':

The installation begins. You see a progress bar. It stays a long time on "Running package scripts" because it is installing XCode Command Line Tools, MacPort, Tesseract OCR, Java Runtime Environment and finally Datashare.

You can see what it actually does by typing command+L, it will open a window which logs every action made.

In the end, you should see this screen:

What is Datashare?

Who uses it?

Curious to know more about how we use Datashare?

Where can I see it in action?

Can I run it on my server?

Can I customize Datashare?

Translations

Once Datashare is installed, go to "Finder", then "Applications", and double-click on "Datashare".

A Terminal window called 'Datashare.command' opens and describes the technical operations going on during the opening.

Keep this Terminal window open as long as you use Datashare.

Open the Windows main menu at the left of the bar at the bottom of your computer screen and click on 'Datashare'. (The numbers after 'Datashare' just indicate which version of Datashare you installed.)

A window called 'Terminal' will have opened, showing the progress of opening Datashare. Do not close this black window as long as you use Datashare.

Keep this Terminal window open as long as you use Datashare.

Find your Datashare folder on your Mac

Open your Mac's 'Finder' by clicking on the blue smiling icon in your Mac's 'Dock':

On the menu bar at the top of your computer, click 'Go'. Click on 'Home' (the house icon).

You will see a folder called 'Datashare':

If you want to quickly access it in the future, you can drag and drop it in 'Favorites' on the left of this window:

Add documents in your Datashare folder

Copy or place the documents you want to have in Datashare in this Datashare folder.

Launch Datashare

Open your Applications. You should see Datashare. Double click on it:

Datashare opens in your default internet browser. Click 'Tasks':

Click the 3rd tab 'Analyze your documents':

Standard (Windows 7 Service Pack 2 or newer version)

Uninstall any prior standard version

Download Datashare

The file "datashare-X.Y.Z.exe" is now downloaded. Double click on the name of the file in order to execute it.

Allow Datashare to install

As Datashare is not signed, this popup asks for your permission. Don't click 'Don't run' but click 'More info':

Click 'Run anyway':

It asks if you want to allow the app to make changes to your device. Click 'Yes':

Install Datashare

On the Installer Wizard, as you need to download and install OpenJDK11 if it is not installed on your device, click 'Install':

The following windows with progress bars will be displayed:

Choose a language and click 'OK':

Install Tesseract OCR

To install Tesseract OCR, click the following buttons on the Installer Wizard's windows:

Untick 'Show README' and click 'Finish':

Finally, click "Close" to close the installer of TesseractOCR.

Install Datashare.jar

It now downloads the back end and the front end, Datashare.jar:

When it is finished, click 'Close':

When you open your desktop, you will see a folder called 'Datashare Data'. Move or copy and paste the documents you want to add to Datashare to this folder:

Once Datashare has opened, click on 'Analyze documents' on the top navigation bar in Datashare:​

Currently, only a .deb package for Debian/Ubuntu is provided.

1. Download Datashare

Save the Debian package as a file

2. Install the package

$ sudo apt install /dir/to/debian/package/datashare-dist_7.2.0_all.deb

3. Run Datashare with:

$ datashare

Web modes

Those two modes are the only one who create

In local mode and embedded mode, Datashare provides a self-contained software application that users can install and run on their own local machines. The software allows users to search into their documents within their own local environments, without relying on external servers or cloud infrastructure. This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.

In server mode, Datashare operates as a centralized server-based system. Users can access to the platform through a web interface, and the documents are stored and processed on Datashare's servers. This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.

Comparaison between modes

The running modes offer advantages and limitations. This matrix summarizes the differences:

When running Datashare in local mode, users can choose to use embedded services (like ElasticSearch, SQLITE, in-memory key/value store) on the same JVM than Datashare. This variant of the local mode is called "embedded mode" and allows user to run Datashare without having to setup any additional software. The embedded mode is used by default.

CLI mode

In cli mode, Datashare starts without a web server and allow user to perform task over their documents. This mode can be used in conjunction both with local and server modes, while allowing users to distribute heaving task between several servers.

Daemon modes

Those modes are intended to be used for action that requires to "wait" for pendings tasks.

In batch download mode, the daemon wait for a user to request a batch download of documents. When a request is receive, the daemon start a task to download the document matching the user search, a bundle them into a zip file.

In batch search mode, the daemon wait for a user to request a batch search of documents. To create a batch search, users must go through the dedicated form on Datashare where they can to upload a list of search terms (in CSV format). The daemon will then start a task to search all matching document and store every occurences in the database.

How to change modes

Datashare is shipped as a single executable, with all modes available. As previously mentioned, the default mpode is the embedded mode. Yet when starting Datashare in command line, you can explicitely specify the running mode. For instance on Ubuntu/Debian:

datashare \
  # Switch to SERVER mode
  --mode SERVER \
  # Dummy session filter to creates ephemeral users
  --authFilter org.icij.datashare.session.YesCookieAuthFilter \
  # Name of the default project for every user
  --defaultProject local-datashare \
  # URI of Elasticsearch
  --elasticsearchAddress http://elasticsearch:9200 \
  # URI of Redis 
  --redisAddress redis://redis:6379 \
  # store user sessions in Redis.
  --sessionStoreType REDIS

Add documents

1. To add your documents in Datashare, click 'Tasks' in the left menu:

2. Click 'Analyze your documents':

3. Click 'Add documents' so Datashare can extract the texts from your files:

Options when adding documents

You can:

• Select the specific folder or sub-folder containing the documents you want to add.

• Extract text also from images/PDFs (OCR). Be aware the indexing can be up to 10 times longer.

• Skip already indexed files.

Two extraction tasks are now running: the first is the scanning of your Datashare folder which sees if there are new documents to analyze (ScanTask). The second is the indexing of these files (IndexTask):

It is not possible to 'Find people, organizations and locations' while of these two tasks is still running.

When tasks are done, you can start exploring documents by clicking 'Search' in the left menu but you won't have the named entities (names of people, organizations and locations) yet. To have these, follow the steps below.

Extract names of people, organizations and locations

1. After the text is extracted, you can launch named entities recognition by clicking the button 'Find people, organizations and locations'.

2. In the window below, you are asked to choose between finding Named Entities or finding email addresses (you cannot do both simultaneously, you need to do one after the other, no matter the order):

You can now see running tasks and their progress. After they are done, you can click 'Clear done tasks' to stop displaying tasks that are completed.

3. You can search your indexed documents without having to wait for all tasks to be done. To access your documents, click 'Search':

Extract email addresses

To extract email addresses in your documents:

• Re-click on 'Find people, organizations, locations and email addresses' (in Tasks (left menu) > Analyze your documents)

• Click the second radio button 'Find email addresses':

Start Datashare by launching it from the command-line:

The CLI stages are primarly intented to be run for an instance of Datashare that uses non-embedded resources (ElasticSearch, database, key/value memory store). This allows you to distribute heaving tasks between servers.

1. SCAN

This is the first step to add documents to Datashare from the command-line. The SCAN stage allows you to queue all the files that need to be indexed (next step). Once this task is done, you can move to the next step. This stage cannot be distributed.

datashare --mode CLI \  
  # Select the SCAN stage
  --stage SCAN \
  # Where the document are located
  --dataDir /path/to/documents \
  # Store the queued files in Redis
  --dataBusType REDIS \
  # URI of Redis 
  --redisAddress redis://redis:6379

2. INDEX

datashare --mode CLI \
  # Select the INDEX stage
  --stage INDEX \
  # Where the document are located
  --dataDir /path/to/documents \
  # Store the queued files in Redis
  --dataBusType REDIS \
  # URI of Elasticsearch
  --elasticsearchAddress http://elasticsearch:9200 \
  # Enable OCR \
  --ocr true
  # URI of Redis 
  --redisAddress redis://redis:6379

3. NLP

Once a document is available for search (stored in ElasticSearch), you can use the NLP stage to extract named entities from the text. This process will not only create named entities mentions in ElasticSearch, it will mark every analyzed document with the corresponding NLP pipeline (CORENLP by default). In other words, the process is idempotent and can be paralelized as well on several servers.

datashare --mode CLI \
  # Select the NLP stage
  --stage NLP \
  # Use CORENLP to detect named entities
  --nlpp CORENLP \
  # URI of Elasticsearch
  --elasticsearchAddress http://elasticsearch:9200 

To be able to perform OCR, Datashare uses an open source technology called Apache Tesseract. When Tesseract extracts text from images, it uses "language packages" especially trained for each specific languages. Unfortunately, those packages can be heavy and to ensure a lightweight installation of Datashare, the installer doesn't use them all by default. In the case Datashare informs you of a missing package, this guide will explain you how to manually install it on your system.

Install packages on Linux

To add ocr languages on linux, simply use the following command:

sudo apt install tesseract-ocr-[lang]

Where `[lang]` is can be :

• all if you want to install all languages

Install packages on Mac

With MacPorts (default)

First, you must check that MacPort is installed on your computer. Please run in a Terminal:

port version

You should see an output similar to this:

If MacPort is installed on your computer, you should be able to add the missing Tesseract language package with the following command (for German):

port install tesseract-deu

Once the installation is done, simply close and restart Datashare to be able to use the newly installed packages.

With Homebrew

If Homebrew was already present on your system when Datashare was installed, Datashare used it to install Tesseract and its language packages. Because Homebrew doesn't package each Tesseract language individually, all languages are already supported by your system. In other words, you have nothing to do!

If you want to check if Homebrew is installed, run the following command in a Terminal:

brew -v

You should see an output similar to this:

Install languages on Windows

*Additionnal languages can be also added during Tesseract's installation.

The list of installed languages can be checked with Windows command prompt or Powershell with the commandtesseract --list-langs.

Datashare has to be restarted after the language installation.

Prerequisites

Get neo4j up and running

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the 'Other platforms and versions' button when downloading to access versions if necessary.

Add entities

If your project contains email documents, make sure to also extract email addresses.

Next step

1. You can find a folder called 'Datashare' in your home directory.

2. Move the documents you want to add to Datashare into this folder.

3. Open Datashare to extract text and eventually find people, organizations and locations in your documents.

Prerequisites

Datashare platform is designed to function effectively by utilizing a combination of various services. To streamline the development and deployment workflows, Datashare relies on the use of Docker containers. Docker provides a lightweight and efficient way to package and distribute software applications, making it easier to manage dependencies and ensure consistency across different environments.

Starting Datashare with a single container

docker run --mount src=$HOME/Datashare,target=/home/datashare/data,type=bind -p 8080:8080 icij/datashare:11.1.9 --mode EMBEDDED

Make sure the Datashare folder exists in your homedir or this command will fail. This is an example about how to use Datashare with Docker, data will not be persisted.

Starting Datashare with multiple containers

Within Datashare, Docker Compose can play a significant role in enabling the setup of separated and persistent services for essential components such as the database (PostgreSQL), the search index (Elasticsearch), and the key-value store (Redis).

By utilizing Docker Compose, you can define and manage multiple containers as part of a unified service. This allows for seamless orchestration and deployment of interconnected services, each serving a specific purpose within the Datashare ecosystem.

Specifically, Docker Compose allows you to configure and launch separate containers for PostgreSQL, Elasticsearch, and Redis. These containers can be set up in a way that ensures their data is persistently stored, meaning that any information or changes made to the database, search index, or key-value store will be retained even if the containers are restarted or redeployed.

This separation of services using Docker Compose provides several advantages. It enhances modularity, scalability, and maintainability within the Datashare platform. It allows for independent management and scaling of each service, facilitating efficient resource utilization and enabling seamless upgrades or replacements of individual components as needed.

version: "3.7"
services:

  datashare:
    image: icij/datashare:18.1.3
    hostname: datashare
    ports:
      - 8080:8080
    environment:
      - DS_DOCKER_MOUNTED_DATA_DIR=/home/datashare/data
    volumes:
      - type: bind
        source: ${HOME}/Datashare
        target: /home/datashare/data
      - type: volume
        source: datashare-models
        target: /home/datashare/dist
    command: >-
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password 
      --mode LOCAL
      --tcpListenPort 8080
    depends_on:
      - postgresql
      - redis
      - elasticsearch

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
    restart: on-failure
    volumes:
      - type: volume
        source: elasticsearch-data
        target: /usr/share/elasticsearch/data
        read_only: false
    environment:
      - "http.host=0.0.0.0"
      - "transport.host=0.0.0.0"
      - "cluster.name=datashare"
      - "discovery.type=single-node"
      - "discovery.zen.minimum_master_nodes=1"
      - "xpack.license.self_generated.type=basic"
      - "http.cors.enabled=true"
      - "http.cors.allow-origin=*"
      - "http.cors.allow-methods=OPTIONS, HEAD, GET, POST, PUT, DELETE"

  redis:
    image: redis:4.0.1-alpine
    restart: on-failure

  postgresql:
    image: postgres:12-alpine
    environment:
      - POSTGRES_USER=datashare
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=datashare
    volumes:
      - type: volume
        source: postgresql-data
        target: /var/lib/postgresql/data

volumes:
  datashare-models:
  elasticsearch-data:
  postgresql-data:

Open a terminal or command prompt and navigate to the directory where you saved the docker-compose.yml file. Then run the following command to start the Datashare service:

docker-compose up -d

The -d flag runs the containers in detached mode, allowing them to run in the background.

Docker Compose will pull the necessary Docker images (if not already present) and start the containers defined in the YAML file. Datashare will take a few seconds to start. You can check the progression of this opperation with:

docker-compose logs -f datashare

Once the containers are up and running, you can access the Datashare service by opening a web browser and entering http://localhost:8080. This assumes that the default port mapping of 8080:8080 is used for the Datashare container in the YAML file.

That's it! You should now have the Datashare service up and running, accessible through your web browser. Remember that the containers will continue to run until you explicitly stop them.

To stop the Datashare service and remove the containers, you can run the following command in the same directory where the docker-compose.yml file is located:

docker-compose down

This will stop and remove the containers, freeing up system resources.

In server mode, Datashare operates as a centralized server-based system. Users can access to the platform through a web interface, and the documents are stored and processed on Datashare's servers. This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.

Launch configuration

Datashare is launched with --mode SERVER and you have to provide:

• the external elasticsearch index address elasticsearchAddress

• a Redis store address redisAddress

• a Redis data bus address messageBusAddress

• the host of Datashare (used to generate batch search results URLs) rootHost

• an authentication mechanism and its parameters

Example:

docker run -ti ICIJ/datashare:version --mode SERVER \
    --redisAddress redis://my.redis-server.org:6379 \
    --elasticsearchAddress https://my.elastic-server.org:9200 \
    --messageBusAddress my.redis-server.org \
    --dataSourceUrl jdbc:postgresql://db-server/ds-database?user=ds-user&password=ds-password \
    --rootHost https://my.datashare-server.org
    # ... +auth parameters (see authentication providers section)

Add plugins to Datashare (front-end)

1. Go to "Settings":

1. Click "Plugins":

1. Choose the plugin you want to add and click "Install now":

If you want to install a plugin from an URL, click "Install plugin from URL".

1. Your plugin is installed.

Refresh your page to see your new plugin activated in Datashare.

Add extensions to Datashare (back-end)

1. Go to "Settings":

1. Click "Extensions":

1. Choose the extension you want to add and click "Install now":

If you want to install an extension from an URL, click "Install extension from URL".

1. Your extension is installed.

Restart Datashare to see your new extension activated in Datashare.

Update plugin or extension with latest version

When a newer version of a plugin or extension is available, you can click on the "Update" button to get the latest version.

After that, if it is a plugin, refresh your page to activate the latest version.

If it is an extension, restart Datashare to activate the latest version.

Create your own plugin or extension

People who code can create their own plugins and extensions by following these steps:

This is likely to be changed in the near future, but in the meantime, you can extract named entities using the command-line interface.

What's happening here:

• We tell Datashare to use the elasticsearch service

• Datashare will pull documents from ElasticSearch directly

• Up to 2 documents will be analyzed in parallel

• Datashare will use the CORENLP pipeline

Datashare will use the output queue from the previous INDEX stage (by default extract:queue:nlp in Redis) that contains all the document ids to be analyzed.

The first time you run this command you will have to wait a little bit because Datashare need to download CORENLP's models which can be big.

You can also use chain the 3 stages altogether:

The added ENQUEUEIDX stage will read Elasticsearch index, find all documents that have not already been analyzed by the CORENLP NER pipeline, and put the ids of those documents into the extract:queue:nlp queue.

Basic authentication is a simple protocol that uses the HTTP headers and the browser to authenticate users. User credentials are sent to the server in the header Authorization with user:password base64 encoded:

Authorization: Basic dXNlcjpwYXNzd29yZA==

It is secure as long as the communication to the server is encrypted (with SSL for example).

On the server side, you have to provide a database user inventory. You can launch datashare first with the full database URL, then datashare will automatically migrate your database schema. Datashare supports SQLite and PostgreSQL as back-end databases. SQLite is not recommended for a multi-user server because it cannot be multithreaded, so it will introduce contention on users' DB SQL requests.

Then you have to provision users. The passwords are sha256 hex encoded (for example with bash):

$ echo -n bar | sha256sum
fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9  -

Then you can insert the user like this in your database:

$ psql datashare
datashare=> insert into user_inventory (id, email, name, provider, details) values ('fbar', 'foo@bar.com', 'Foo Bar', 'my_company', '{"password": "fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9", "groups_by_applications":{"datashare":["local-datashare"]}}');

If you use other indices, you'll have to include them in the group_by_applications, but local-datashare should remain. For exammple if you use myindex:

$ psql datashare
datashare=> insert into user_inventory (id, email, name, provider, details) values ('fbar', 'foo@bar.com', 'Foo Bar', 'my_company', '{"password": "fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9", "groups_by_applications":{"datashare":["myindex", "local-datashare"]}}');

Then when accessing Datashare, you should see this popup:

Example

Here is an example of launching Datashare with Docker and the basic auth provider filter backed in database:

docker run -ti ICIJ/datashare --mode SERVER \
    --batchQueueType REDIS \
    --dataSourceUrl 'jdbc:postgresql://postgres/datashare?user=<username>&password=<password>' \
    --sessionStoreType REDIS \
    --authFilter org.icij.datashare.session.BasicAuthAdaptorFilter \
    --authUsersProvider org.icij.datashare.session.UsersInDb

This is likelly to be changed in the near future, but in the meantime, you can still add documents to Datashare using the command-line interface.

Here is a simple command to scan a directory and index its files:

What's happening here:

• The SCAN stage feeds a queue in memory with file to add

• The INDEX stage pulls files from the queue to add them to ElasticSearch

• We tell Datashare to use the elasticsearch service

• Files to add are located in /home/datashare/Datashare/ which is a directory mounted from the host machine

Alternativly, you can do this in two separated phases, as long as you tell Datashare to store the queue in a shared resource. Here, we use the redis:

Once the opperation is done, we can easily check the content of queue created by Datashare in redis. In this example we only display the 20 first files in the datashare:queue:

Once the indexing is done, Datashare will exit gracefully and your document will already be visible on Datashare.

Sometimes you will face the case where you have an existing index, and you want to index additional documents inside your working directory without processing every document again. It can be done in two steps :

• Scan the existing ElasticSearch index and gather document paths to store it inside a report queue

• Scan and index (with OCR) the documents in the directory, thanks to the previous report queue, it will skip the paths inside of it

Authentication with Datashare in server mode is the most impacting choice that has to be made. It can be one of the followings:

• basic authentication with credentials stored in database (PostgreSQL)

• basic authentication with credentials stored in Redis

• OAuth2 with credentials provided by an identity provider (KeyCloak for example)

• dummy basic auth to accept any user (⚠️ if the service is exposed to internet, it will leak your documents)

Prerequisites

Datashare platform is designed to function effectively by utilizing a combination of various services. To streamline the development and deployment workflows, Datashare relies on the use of Docker containers. Docker provides a lightweight and efficient way to package and distribute software applications, making it easier to manage dependencies and ensure consistency across different environments.

Starting Datashare with multiple containers

Within Datashare, Docker Compose can play a significant role in enabling the setup of separated and persistent services for essential components. By utilizing Docker Compose, you can define and manage multiple containers as part of a unified service. This allows for seamless orchestration and deployment of interconnected services, each serving a specific purpose within the Datashare ecosystem.

Specifically, Docker Compose allows you to configure and launch separate containers for PostgreSQL, Elasticsearch, and Redis. These containers can be set up in a way that ensures their data is persistently stored, meaning that any information or changes made to the database, search index, or key-value store will be retained even if the containers are restarted or redeployed.

This separation of services using Docker Compose provides several advantages. It enhances modularity, scalability, and maintainability within the Datashare platform. It allows for independent management and scaling of each service, facilitating efficient resource utilization and enabling seamless upgrades or replacements of individual components as needed.

version: "3.7"
services:

  # This is the main Datashare service that serves the web interface. 
  # Here it is configured with a "dummy" authentication backend which 
  # creates epehemeral user sessions.
  datashare_web:
    image: icij/datashare:18.1.3
    hostname: datashare
    ports:
      - 8080:8080
    environment:
      - DS_DOCKER_MOUNTED_DATA_DIR=${HOME}/Datashare
    volumes:
      - type: bind
        source: ${HOME}/Datashare
        target: /home/datashare/Datashare
    depends_on:
      postgresql:
        condition: service_healthy
      redis:
        condition: service_healthy
      elasticsearch:
        condition: service_healthy
    command: >-
      --mode SERVER
      --dataDir /home/datashare/Datashare
      --pluginsDir /home/datashare/plugins
      --extensionsDir /home/datashare/extensions
      --authFilter org.icij.datashare.session.YesCookieAuthFilter
      --busType REDIS
      --batchQueueType REDIS
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password 
      --defaultProject secret-project
      --elasticsearchAddress http://elasticsearch:9200      
      --messageBusAddress redis://redis:6379
      --queueType REDIS
      --redisAddress redis://redis:6379  
      --rootHost http://localhost:8080
      --sessionStoreType REDIS
      --sessionTtlSeconds 43200
      --tcpListenPort 8080

  # We use a service to create the "secret-project". In theory you only need 
  # to run it the first time you start Datashare. 
  datashare_create_project:
    image: icij/datashare:18.1.3
    restart: no
    depends_on:
      elasticsearch:
        condition: service_healthy
    command: >-
      --defaultProject secret-project 
      --mode CLI 
      --stage INDEX 
      --elasticsearchAddress http://elasticsearch:9200

  # This service starts a deamon that wait for background tasks
  # so it can run them (and save them in the database).
  datashare_task:
    image: icij/datashare:18.1.3
    depends_on:
      - datashare_web
    command: >-
      --mode TASK_RUNNER
      --batchQueueType REDIS
      --batchThrottleMilliseconds 500
      --busType REDIS
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password
      --defaultProject secret-project 
      --elasticsearchAddress http://elasticsearch:9200  
      --queueType REDIS
      --redisAddress redis://redis:6379
      --scrollSize 100  
      
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
    restart: on-failure
    volumes:
      - type: volume
        source: elasticsearch-data
        target: /usr/share/elasticsearch/data
        read_only: false
    environment:
      - "http.host=0.0.0.0"
      - "transport.host=0.0.0.0"
      - "cluster.name=datashare"
      - "discovery.type=single-node"
      - "discovery.zen.minimum_master_nodes=1"
      - "xpack.license.self_generated.type=basic"
      - "http.cors.enabled=true"
      - "http.cors.allow-origin=*"
      - "http.cors.allow-methods=OPTIONS, HEAD, GET, POST, PUT, DELETE"
    healthcheck:
      test: ["CMD-SHELL", "curl --silent --fail elasticsearch:9200/_cluster/health || exit 1"]

  postgresql:
    image: postgres:12-alpine
    environment:
      - POSTGRES_USER=datashare
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=datashare
      # This is needed by the heathcheck command
      # @see https://stackoverflow.com/a/60194261
      - PGUSER=datashare
    volumes:
      - type: volume
        source: postgresql-data
        target: /var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready", "-U", "datashare", "-d", "datashare"]
  
  redis:
    image: redis:4.0.1-alpine
    restart: on-failure
    volumes:
      - type: volume
        source: redis-data
        target: /data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli", "--raw", "incr", "ping"]

volumes:
  datashare-batchdownload-dir:
  elasticsearch-data:
  postgresql-data:
  redis-data:

Open a terminal or command prompt and navigate to the directory where you saved the docker-compose.yml file. Then run the following command to start the Datashare service:

docker-compose up -d

The -d flag runs the containers in detached mode, allowing them to run in the background.

Docker Compose will pull the necessary Docker images (if not already present) and start the containers defined in the YAML file. Datashare will take a few seconds to start. You can check the progression of this opperation with:

docker-compose logs -f datashare_web

Once the containers are up and running, you can access the Datashare service by opening a web browser and entering http://localhost:8080. This assumes that the default port mapping of 8080:8080 is used for the Datashare container in the YAML file.

To stop the Datashare service and remove the containers, you can run the following command in the same directory where the docker-compose.yml file is located:

docker-compose down

This will stop and remove the containers, freeing up system resources.

Add documents to Datashare

Extract named entities

Create the graph

1. Open the 'Projects' page and select your project:

1. Create the graph by clicking on the 'Create graph' button inside the neo4j widget:

You will see a new import task running:

When the graph creation is complete, 'Graph statistics' will reflect the number of documents and entities nodes found in the graph:

Update the graph

When new documents or entities are added or modified inside Datashare, you will need to update the neo4j graph to reflect these changes.

To update the graph click on the 'Update graph' button inside the neo4j widget:

To detect whether a graph update is needed you can compare the number of documents found inside Datashare to the number found in the 'Graph statistics' and run an update in case of mismatch:

The update will always add missing nodes and relationships, update existing ones if they were modified, but will never delete graph nodes or relationships.

Next steps

Basic authentication is a simple protocol that uses the HTTP headers and the browser to authenticate users. User credentials are sent to the server in the header Authorization with user:password base64 encoded:

It is secure as long as the communication to the server is encrypted (with SSL for example).

On the server side, you have to provide a user store for Datashare. For now we are using a Redis data store.

So you have to provision users. The passwords are sha256 hex encoded. For example using bash:

Then insert the user like this in Redis:

If you use other indices, you'll have to include them in the group_by_applications, but local-datashare should remain. For exammple if you use myindex:

Then you should see this popup:

Example

Here is an example of launching Datashare with Docker and the basic auth provider filter backed in Redis:

Install the neo4j plugin

Configure the neo4j plugin

1. Go to "Settings":

2. Make sure the following settings are properly set:

• Neo4j Host should be localhost or the address where your neo4j instance is running

• Neo4j Port should be the port where your neo4j instance is running (7687 by default)

• Neo4j User should be set to your neo4j user name (neo4j by default)

• Neo4j Password should only be set if your neo4j user is using password authentication

3. When running Neo4j Community Edition, set the Neo4j Single Project value. In community edition, the neo4j DBMS is restricted to a single database. Since Datashare supports multiple projects, you must set the Neo4j Single Project with the name of the project which will use neo4j plugin. Other projects won't be able to use the neo4j plugin.

4. Restart Datashare to apply the changes

5. You should be able to see the neo4j widget in your project page, after a little while its status should be RUNNING:

Next step

Prerequisites

Get neo4j up and running

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the 'Other platforms and versions' button when downloading to access versions if necessary.

Add entities

If your project contains email documents, make sure to run the EMAIL pipeline together with regular NLP pipeline. To do so add set the follow nlpp flag to --nlpp CORENLP,EMAIL.

Next step

You can have a dummy authentication that always accepts basic auth. So you should see this popup:

But then you can enter whatever user or password, it will enter Datashare.

Example

docker run -ti ICIJ/datashare -m SERVER \
  --dataDir /home/dev/data \
    --batchQueueType REDIS \
    --dataSourceUrl 'jdbc:postgresql://postgres/datashare?user=dstest&password=test'\
    --sessionStoreType REDIS \
    --authFilter org.icij.datashare.session.YesBasicAuthFilter

This is the default authentication mode: if not provided in CLI it will be selected. With OAuth2 you will need a third-party authorization service. The diagram below describes the workflow:

Example

docker run -ti ICIJ/datashare:version --mode SERVER \
    --oauthClientId 30045255030c6740ce4c95c \
    --oauthClientSecret 10af3d46399a8143179271e6b726aaf63f20604092106 \
    --oauthAuthorizeUrl https://my.oauth-server.org/oauth/authorize \
    --oauthTokenUrl https://my.oauth-server.org/oauth/token \
    --oauthApiUrl https://my.oauth-server.org/api/v1/me.json \
    --oauthCallbackPath /auth/callback

Integration with KeyCloak

Run the neo4j extension CLI

The neo4j related features are added to the DatashareCLI through the extension mechanism. In order to run the extended CLI, the Java CLASSPATH must be extended with the path of the datashare-extension-neo4j jar. By default, this jar is located in /home/datashare/extensions, so the CLI will be run as following:

docker compose exec \
  -e CLASSPATH=/home/datashare/extensions/* \
  datashare_web /entrypoint.sh \
  --mode CLI \
  --ext neo4j \
  ... 

Create the graph

In order to create the graph, run the --fullImport command for your project:

docker compose exec \
  -e CLASSPATH=/home/datashare/extensions/* \
  datashare_web /entrypoint.sh \
  --mode CLI \
  --ext neo4j \
  --full-import \
  --project secret-project

the CLI will display the import task progress and log import related information.

Update the graph

When new documents or entities are added or modified inside Datashare, you will need to update the neo4j graph to reflect these changes.

To update the graph, you can just re-run the full export:

docker compose exec \
  -e CLASSPATH=/home/datashare/extensions/datashare-extension-*.jar \
  datashare_web /entrypoint.sh \
  --mode CLI \
  --ext neo4j \
  --full-import \
  --project secret-project

The update will always add missing nodes and relationships, update existing ones if they were modified, but will never delete graph nodes or relationships.

To detect whether a graph update is needed, open the 'Projects' page and select your project:

compare the number of documents and entities found inside Datashare:

to the numbers found in the 'Graph statistics' and run an update in case of mismatch:

Next steps

Search in documents

If not collapsed yet, to collapse the left menu in order to gain room, click the 'hamburger menu':

2. Search for specific documents. Type terms in the search bar, press Enter or click 'Search':

IMPORTANT:

3. You can search in specific fields like tags, title, author, recipient, content, path or thread ID. Click 'All fields' and select your choice in the dropdown menu:

Choose between views (list, grid, table)

Select the view on the top right.

• List:

• Grid:

• Table:

Search within a document

Once a document is opened, you can search for terms in this document:

• Press Command (⌘) + F (on Mac) or Control + F (on Windows and Linux) or click on the search bar above your Extracted Text

• Type what you search for

• Press ENTER to go from one occurrence to the next one

• Presse SHIFT + ENTER to go from one occurrence to the previous one

This also counts the number of occurrences of your searched terms in this document:

Double quotes for exact phrase

To have all documents mentioning an exact phrase, you can use double quotes. Use straight double quotes ("example"), not curly double quotes (“example”).

Example: "Alicia Martinez’s bank account in Portugal"

OR (or space)

To have all documents mentioning all or one of the queried terms, you can use a simple space between your queries or 'OR'. You need to write 'OR' with all letters uppercase.

Example: Alicia Martinez

Same search: Alicia OR Martinez

AND (or +)

To have all documents mentioning all the queried terms, you can use 'AND' between your queried words. You need to write 'AND' with all letters uppercase.

Example: Alicia AND Martinez

Same search: +Alicia +Martinez

NOT (or ! or -)

To have all documents NOT mentioning some queried terms, you can use 'NOT' before each word you don't want. You need to write 'NOT' with all letters uppercase.

Example: NOT Martinez

Same search: !Martinez

Same search: -Martinez

Please note that you can combine operators

Parentheses should be used whenever multiple operators are used together and you want to give priority to some.

Example: ((Alicia AND Martinez) OR (Delaware AND Pekin) OR Grey) AND NOT "parking lot"

Wildcards

If you search faithf?l, the search engine will look for all words with all possible single character between the second f and the l in this word. It also works with * to replace multiple characters.

Example: Alicia Martin?z

Example: Alicia Mar*z

Fuzziness

You can set fuzziness to 1 or 2. It corresponds to the maximum number of operations (insertions, deletions, substitutions and transpositions) on characters needed to make one term match the other.

kitten -> sitten (1 substitution (k turned into s) = fuzziness is 1)

kitten -> sittin (2 substitutions (k turned into s and e turned into i) = fuzziness is 2)

Example: quikc~ brwn~ foks~ (as the default edit distance is 2, this query will catch all quick, quack, quock, uqikc, etc. as well as brown, folks, etc.)

Example: Datashare~1 (this query will catch Datasahre, Dqtashare, etc.)

Proximity searches

When you type an exact phrase (in double quotes) and use fuzziness, then the meaning of the fuzziness changes. Now, the fuzziness means the maximum number of operations (insertions, deletions, substitutions and transpositions) on terms needed to make one phrase match the other.

Examples:

"the cat is blue" -> "the small cat is blue" (1 insertion = fuzziness is 1)

"the cat is blue" -> "the small is cat blue" (1 insertion + 2 transpositions = fuzziness is 3)

Example: "fox quick"~5 (this query will catch "quick brown fox", "quick brown car thin fox" or even "quick brown car thin blue tree fox"

Boosting operators

Use the boost operator ^ to make one term more relevant than another. For instance, if we want to find all documents about foxes, but we are especially interested in quick foxes:

Example: quick^2 fox

The default boost value is 1, but can be any positive floating point number. Boosts between 0 and 1 reduce relevance. Boosts can also be applied to phrases or to groups:

Example: "john smith"^2 (foo bar)^4

Regular expressions (Regex)

1. You can use Regex in Datashare. Regular expressions (Regex) in Datashare need to be written between 2 slashes.

Example: /.*..*@.*..*/

The example above will search for any expression which is structured like an email address with a dot between two expressions before the @ and a dot between two expressions after the @ like in 'first.lastname@email.com' for instance.

2. Regex can be combined with standard queries in Datashare :

Example: ("Ada Lovelace" OR "Ado Lavelace") AND paris AND /.*..*@.*..*/

3. You need to escape the following characters by typing a backslash just before them (without space):‌ # @ & < > ~

Example: /.*..*@.*..*/ (the @ was escaped by a backslash \ just before it)

We encourage you to use the AND operator to work around this limitation and make sure you can make your search.

If you're looking for French International Bank Account Number (IBAN) that can or cannot contain space and contain FR followed by numbers and/or letters (it could be FR7630001007941234567890185 ou FR76 3000 4000 0312 3456 7890 H43 for example), you can then search for:

Example: /FR[0-9]{14}[0-9a-zA-Z]{11}/ OR (/FR[0-9]{2}.*/ AND /[0-9]{4}.*/ AND /[0-9a-zA-Z]{11}.*/)

1. Here are a few examples of useful Regex:

• You can search for /Dimitr[iyu]/ instead of searching for Dimitri OR Dimitry OR Dimitru. It will find all the Dimitri, Dimitry or Dimitru.

• You can search for /Dimitr[^yu]/ if you want to search all the words which begin with Dimitr and do not end with either y nor u.

• You can search for /Dimitri<1-5>/ if you want to search Dimitri1, Dimitri2, Dimitri3, Dimitri4 or Dimitri5.

1. Other common Regex examples:

• phone numbers with "-" and/or country code like +919367788755, 8989829304, +16308520397 or 786-307-3615 for instance: /[\+]?[(]?[0-9]{3}[)]?[-\s.]?[0-9]{3}[-\s.]?[0-9]{4,6}/

• credit cards: /(?:4[0-9]{12}(?:[0-9]{3})?|[25][1-7][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35[0-9]{3})[0-9]{11})/

(Advanced) Searches using metadata fields

To find the list of existing metadata fields, go to a document's 'Tags and details' tab, click 'Show more details'.

When you hover the lines, you see a magnifying glass on each line. Click on it and Datashare will look for this field. Here is the one for content language:

Here is the one for 'indexing date' (also called extraction date here) for instance:

So for example, if you are looking for documents that:

• contains term1, term2 and term3

• and were created after 2010

you can use the 'Date' filter or type in the search bar:

term1 AND term2 AND term3 AND metadata.tika_metadata_creation_date:>=2010-01-01

Explanations:

• 'metadata.tika_metadata_creation_date:' means that we filter with creation date

• '>="'means 'since January 1st included'

• '2010-01-01' stands for January 2010 and the search will include January 2010

For other searches:

• '<' will mean 'strictly after (with January 1st excluded)'

• nothing will mean 'at this exact date'

You can search for numbers in a range. Ranges can be specified for date, numeric or string fields amont the ones you can find by clicking the magnifying glass when you hover the fields in a document's tab 'Tags and Details'.

Install the neo4j plugin

Install the neo4j plugin using the Datashare CLI so that users can access it from the frontend:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --pluginInstall datashare-plugin-neo4j-graph-widget 

Installing the plugin install the datashare-plugin-neo4j-graph-widget plugin inside /home/datashare/plugings and will also install the datashare-extension-neo4j backend extension inside /home/datashare/extensions. These locations can be changed by updating the docker-compose.yml.

Configure the neo4j extension

...
services:
    datashare_web:
      ...
      environment:
        - DS_DOCKER_NEO4J_HOST=neo4j
        - DS_DOCKER_NEO4J_PORT=7687
        - DS_DOCKER_NEO4J_SINGLE_PROJECT=secret-project  # This is for community edition only

If your choose a different neo4j user or set a password for your neo4j user make sure to also set DS_DOCKER_NEO4J_USER and DS_DOCKER_NEO4J_PASSWORD.

When running Neo4j Community Edition, set the DS_DOCKER_NEO4J_SINGLE_PROJECT value. In community edition, the neo4j DBMS is restricted to a single database. Since Datashare supports multiple projects, you must set the DS_DOCKER_NEO4J_SINGLE_PROJECT with the name of the project which will use neo4j plugin. Other projects won't be able to use the neo4j plugin.

Restart Datasahre

After installing the plugin a restart might be needed for the plugin to display:

docker compose restart datashare_web

Next step

Extracted text

In 'Extracted Text', you can read the text of a document as extracted by Datashare:

Please beware that Datashare show named entities by default. This can overwrite some original text with wrong named entities. It is thus important to always verify original text by deactivating named entity overwriting. To do so, please:

• Turn off the toggle button ‘Show named entities’ and read the extracted text

• Check the ‘Preview’ of original document if available

• Check the original document at its original location or by clicking the pink button ‘Download’

****

Search for attachments

If the documents has attachments (technically called 'children documents'), find them at the end of the document. Click their pink button to open them:

To open all the attachments in Datashare, click 'See children documents' in Tags and Details:

****

Search for terms within this document

• Press Command(⌘) + F (on Mac) or Control + F (on Windows and Linux) or click on the search bar above your Extracted Text

• Type what you search for

• Press ENTER to go from one occurrence to the next one

• Presse SHIFT + ENTER to go from one occurrence to the previous one

This also counts the number of occurrences of your searched terms in this document:

Tags & Details

In 'Tags & Details', you can read the document's details. It's all the metadata as they appear in the original file. Please click 'Show more details' to get all metadata:

You can also read the tags you previously wrote for this document, like 'test1', 'test2' and 'test3' in the example below:

You can then search for the documents you tagged:

• Type the tag(s) in the main search bar

• Click 'All fields' and select 'Tags'

• Click 'Search' or press 'Enter'

Named Entities

In 'Named Entities', you can read the name of people, organizations and locations as well as the number of their occurrences in the document:

Please beware that there can still be some errors due to the technology of Named Entity Extraction (NER) on which Datashare relies.

If you run email extraction, you will see a list of the extracted emails:

Preview

In 'Preview', you can read the original document.

'Preview' is available for some formats only.

You can sort documents by:

• relevance (by default): it is a score calculated by the search engine

• indexing date: when you analyzed the document, the day and time you 'put' them in Datashare

• creation date: the day and time the document was created, as it is written in the document's metadata

• size of the documents

• path of the documents

You can also decide the number of documents displayed by page (10, 25, 50 or 100):

Apply filters

On the left column, you can apply filters by ticking them, like 'Portable Document Format (PDF)' in File Types and 'English' in Languages in the example below:

A reminder of the currently applied filters, as well as your queried terms, are displayed at the top of the results' column. You can easily unselect these filters from there by clicking them or clear all of them:

The currently available filters are:

• Projects: if you have more than one project, you can select several of them and run searches in multiple projects at once.

• Starred: If you have starred documents, you can easily find them again.

• Tags: If you wrote some tags, you will be able to select and search for them.

• Recommended by: available only on server (collaborative) mode, this functionality helps you find the document recommended by you and/or others.

• File type: This is the 'Content type' of the file (Word, PDF, JEPG image, etc.) as you can read it in a document's 'Tags & Details'.

• Creation dates: the calendar allows you to select a single creation date or a date range. This is when the document was created as it is noticed in their properties. You can find this in a document's 'Tags & Details'.

• Languages: Datashare detects the main language of each document.

• People / Organizations / Locations: you can selected these named entities and search them.

• Path: This is the location of your documents as it is indicated in your original files (ex: desktop/importantdocuments/mypictures). You can find this in a document's 'Tags & Details'.

• Indexing date: This date corresponds to when you indexed the documents in Datashare.

• Extraction level: This regards embedded documents. The file on disk is level zero. If a document (pictures, etc) is attached or contained in a file on disk, extraction level is “1st”. If a document is attached or contained in a document itself contained in a file on disk, extraction level is “2nd”, etc.

Filters can be combined together and combined with searches in order to refine results.

Filter by named entities

If you have asked Datashare to 'Find people, organizations and locations', you can see names of individuals, organizations and locations in the filters. These are the named entities automatically detected by Datashare.

Search for named entities in the filter's search bar:

Select all of them, one or several of them to filter the documents that mention them:

Use the 'Exclude' button

If you want to select all items except one or several of them, you can use the 'Exclude button'.

It allows you to search for all documents which do not correspond to the filter(s) you selected, that is to say to the currently strikethrough filters.

Contextualize filters

In several filters, you can tick 'Contextualize' : this will update the number of documents indicated in the filters in order to reflect the results. The filter will only count what you selected.

In the example below, the 'Contextualize' checkboxes are not ticked:

After the Contextualize button in Tags filter is ticked:

After the Languages button in Tags filter is ticked:

Clear all filters

To reset all filters at the same time, click 'Clear all':

Recommend a document

Open the document by clicking on its title

• Click the button 'Mark as recommended':

Your recommendation is now displayed on this page and in the left 'Recommended by' filter.

Search recommended documents with the filter

Open the filter untitled 'Recommended by'

Unmark a document as recommended

Open the document and click on "Unmark as recommended".

If you want to search a list of queries in Datashare, instead of doing each of them one by one, you can upload the list directly in Datashare. To do so, you will:

• Create a list of terms that you want to search in the first column of a spreadsheet

• Export the spreadsheet as a CSV (a special format available in any spreadsheet software)

• Upload this CSV in the "new Batch Search" form in Datashare

• Get the results for each query in Datashare - or in a CSV.

Prepare your batch search

Write your queries in a spreadsheet

• Write your queries, one per line and per cell, in the first column of a spreadsheet (Excel, Google Sheets, Numbers, Framacalc, etc.). In the example below, there are 4 queries:

• Do not put line break(s) in any of your cells.

To delete line break(s) in your spreadsheet, you can use the "Find and replace all" functionality. Find all "\n" and replace them all by nothing or a space.

• Write 2 characters minimum in the cells. If one cell contains one character but at least one other cell contains more than one, the cell containing one character will be ignored. If all cells contain only one character, the batch search will lead to 'failure'.

• If you have blank cells in your spreadsheet...

...the CSV (which stand for 'Comma-separated values') will keep these blank cells. It will separate them with semicolons (the 'commas'). You will thus have semicolons in your batch search results (see screenshot below). To avoid that, you need to remove blank cells in your spreadsheet before exporting it as a CSV.

• If there is a comma in one of your cells (like in "1,8 million" in our example above), the CSV will formally put the content of the cell in double quotes in your results and search for the exact phrase in double quotes.

Want to search only on some documents?

For instance, if you want to search only in some documents with certain tag(s), you can write your queries like this: "Paris AND (tags:London OR tags:Madrid NOT tags:Cotonou)".

Use operators in your CSV

Please also note that searches are not case sensitive: if you search 'HeLlo', it will look for all occurrences of 'Hello', 'hello', 'hEllo', 'heLLo', etc. in the documents.

Export your CSV encoded in UTF-8

Export your spreadsheet in a CSV format like this:

• LibreOffice Calc: it uses UTF-8 by default. If not, go to LibreOffice menu > Preferences > Load/Save > HTML Compatibility and make sur the character set is 'Unicode (UTF-8)':

• Google Sheets: it uses UTF-8 by default. Just click "Export to" and "CSV".

• Other spreadsheet softwares: please refer to each software's user guide.

Launch your batch search

• Open Datashare, click 'Batch searches' in the left menu and click 'New batch search' on the top right:

• Type a name for your batch search:

• Upload your CSV:

• Add a description (optional):

• Set the advanced filters ('Do phrase matches', 'Fuzziness' or 'Proximity searches', 'File types' and 'Path') according to your preferences:

What is 'Do phrase matches'?

'Do phrase matches' is the equivalent of double quotes: it looks for documents containing an exact sentence or phrase. If you turn it on, all queries will be search for their exact mention in documents as if Datashare added double quotes around each query.

What is fuzziness?

kitten -> sitten (1 substitution (k turned into s) = fuzziness is 1)

kitten -> sittin (2 substitutions (k turned into s and e turned into i) = fuzziness is 2)

If you search for similar terms (to catch typos for example), use fuzziness.

Example: quikc~ brwn~ foks~ (as the default edit distance is 2, this query will catch all quick, quack, quock, uqikc, etc. as well as brown, folks, etc.)

Example: Datashare~1 (this query will catch Datasahre, Dqtashare, etc.)

What are proximity searches?

When you turn on 'Do phrase matches', you can set, in 'Proximity searches', the maximum number of operations (insertions, deletions, substitutions and transpositions) on terms needed to make one phrase match the other.

“the cat is blue” -> “the small cat is blue” (1 insertion = fuzziness is 1)

“the cat is blue” -> “the small is cat blue” (1 insertion + 2 transpositions = fuzziness is 3)

Example: "fox quick"~5 (this query will catch "quick brown fox", "quick brown car thin fox" or even "quick brown car thin blue tree fox"

• Click 'Add'. Your batch search will appear in the table of batch searches.

Get your results

• Open your batch search by clicking its name:

• You see your results and you can sort them by clicking the column's name. 'Rank' means the order by which each queries would be sorted out if run in Datashare's main search bar. They are thus sorted by relevance score by default.

• You can click on a document's name and it will open it in a new tab:

• You can filter your results by query and read how many documents there are for each query:

You can search for specific queries:

• You can also download your results in a CSV format:

Relaunch your batch search

If you add more and more files in Datashare, you might want to relaunch existing batch search on your new documents too.

Notes:

• In the server collaborative mode, you can only relaunch your own batch searches, not others'.

• The relaunched batch search will apply to your whole corpus, newly indexed documents and previously indexed documents (not only the newly indexed ones).

1. To do so, open the batch search that you'd like to relaunch and click 'Relaunch':

1. Edit the name and the description of your batch search if needed:

1. You can choose to delete the current batch search after relaunching it:

Note: if you're worried about losing your previous results because of an error, we recommend to keep your current batch search (turn off this toggle button) and delete it only after the relaunch is a success.

1. Click 'Submit':

You can see your relaunched batch search running in the batch search's list:

I get a "failure". What does that mean?

Failures in batch searches can be due to several causes.

Click the 'See error' button to open the error window:

The first query containing an error makes the batch search fail and stop.

Check this first failure-generating query in the error window:

We recommend to remove the slash, as well as any reserved characters, and re-run the batch search again.

'elasticsearch: Name does not resolve'

If you have a message which contain 'elasticsearch: Name does not resolve', it means that Datashare can't make Elastic Search, its search engine, work.

Example of a message regarding a problem with ElasticSearch:

SearchException: query='lovelace' message='org.icij.datashare.batch.SearchException: java.io.IOException: elasticsearch: Name does not resolve'

__

'Data too large'

One of your queries can lead to a 'Data too large' error.

It means that this query had too many results or in their results, some documents that were too big to process for Datashare. This makes the search engine fail.

We recommend to remove the query responsible for the error and re-start your batch search without the query which led to the 'Data too large' error.

****

'SearchException: query='AND ada' '

One or several of your queries contains syntax errors.

It means that you wrote one or more of your queries the wrong way with some characters that are reserved as operators (see below).

Datashare stops at the first syntax error. It reports only the first ​error. You might need to check all your queries as some errors can remain after correcting the first one.

They are more likely to happen when 'do phrase matches' toggle button is turned off:

When 'Do phrase matches' is on, syntax error can still happen though:

Here are the most common errors:

- A query starts with AND (all uppercase)

- A query starts with OR (all uppercase)

- A query contains only one double quote or a double quote in a word

- A query starts with or contains tilde (~) inside a term

- A query starts with or contains a caret (^)

- A query contains one slash (/)

- A query uses square brackets ([ ])

Delete your batch search

Open your batch search and click the trash icon:

Then click 'Yes':

Tag a document

• Open the document by clicking on its title

• Click the second tab 'Tags & Details'

• Type your tag

• Press 'Enter'

Tags can contain any character but cannot contain space.

Your new tag is now displayed on this page.

You can add several tags.

Search tags with tag filter

• Open the second filter untitled 'Tags'

• You see the tags by frequency and the number of tagged documents

• You can search using the search bar

• You can select one or multiple tags

Search tags with main search bar

To find all your documents tagged with specific tag(s):

• Type the tag(s) in the main search bar

• Select 'Tags' in the field dropdown menu

• Click 'Search' or press 'Enter'

The results are all the documents tagged with the tag(s) you typed in the search bar.

To find all your tagged documents, whatever the tags:

• Type nothing in the search bar

• Select 'Tags' in the field selector

• Click 'Search'

The results are all the tagged documents.

Delete a tag

Click the cross at the end of the tag that you want to delete.

You need an internet connection to install Datashare.

You also need the internet to find people, organizations and locations in documents the first time you use any new NLP option because the models which find these named entities are downloaded when you ask for finding named entities the first time. Subsequently, you don't need an internet connection to find named entities after.

You don't need internet connection:

• to add documents to Datashare

• to find named entities (except for the first time you use an NLP options - see above)

• to search and explore documents

• to download documents

This allows you to work safely on your documents. No third-party should be able to intercept your data and files while you're working offline on your computer.

You can download a document by going to it on Datashare. Click on the download icon to the right of the screen on on the right of the document's title.

Find these shortcuts in Datashare here:

It will open a window which recalls the shortcuts:

Go to the next / previous document

• Windows / Linux

Control + →

Control + ←

• Mac

Command (⌘) + →

Command (⌘) + ←

Find in document...

• Windows / Linux

Control + F

• Mac

Command (⌘) + F

... and go from one occurrence to the next / previous occurrence

• Go to next occurrence

Enter

or

F3

• Go to previous occurrence

Shift + Enter

or

Shift + F3

Navigate a document's tabs

• Windows / Linux

Control (ctrl) + alt + ⇞ (pageup)

Control (ctrl) + alt + ⇟ (pagedown)

• Mac

Command (⌘) + option (⌥) + ↑ (arrow up)

Command (⌘) + option (⌥) + ↓ (arrow down)

Go back to search results

Once you opened a document, go back to search results:

Esc

The documents and entities graph

After creating a graph from your Datashare project, you will be able to explore it and visualize these kinds of relationships between you project entities:

In the above graph, we can see 3 email document nodes in orange, 3 email address nodes in red, 1 person node in green and 1 location node in yellow. Reading the relationship types on the arrows, we can deduce the following information from the graph:

• shapp@caiso.com emailed 20participants@caiso.com, the sent email has an id starting with f4db344...

• one person named vincent is mentioned inside this email, as well as the california location

• finally, the email also mentions the dle@caiso.com email address which is also mentioned in 2 other email documents (with id starting with 11df197... and 033b4a2...)

If you are not familiar with graph and neo4j, take a look at the following resources:

Graph nodes

The neo4j graph is composed of :Document nodes representing Datashare documents and :NamedEntity nodes representing entities mentioned in these documents.

The :NamedEntity nodes are additionally annotated with their entity types: :NamedEntity:PERSON, :NamedEntity:ORGANIZATION, :NamedEntity:LOCATION, :NamedEntity:EMAIL...

Graph relationships

In most cases, an entity :APPEARS_IN a document, which means that it was detected in the document content. In the particular case of email documents and EMAIL addresses, it is most of the time possible to identify richer relationships from the email metadata, such as who sent (:SENT relationship) and who received (:RECEIVED relationship) the email.

When an :EMAIL address entity is neither :SENT or :RECEIVED, like it is the case in the above graph for dle@caiso.com, it means that the address was mentioned in the email document body.

When a document is embedded inside another document (as an email attachment for instance), the child document is connected to its parent through the :HAS_PARENT relationship.

Create your Datashare project's graph

The creation of a neo4j graph inside Datashare is supported through a plugin. To use the plugin to create a graph, follow these instructions:

After the graph is created, navigate to the 'Projects' page and select your project. You should be able to visualize a new neo4j widget displaying the number of documents and entities found inside the graph:

Access your project's graph

Exporting and importing the graph into your own DB is also useful when you want to perform write operations on your graph without any consequences on Datashare.

With read access to Datashare's neo4j database

Without read access to Datashare's neo4j database

If you can't have read access to the database, you will need to export it and import it into your own neo4j instance (running on your laptop for instance).

Ask for a DB dump

Export your graph from Datashare

In case you don't have access to the DB and can't be provided with a dump, you can export the graph from inside. Be aware that limits might be applied on the size of the exported graph.

To export the graph, navigate to Datashare's 'Projects' page, select your project, select the 'Cypher shell' export format and click the 'Export graph' button:

In case you want to restrict the size of the exported graph, you can restrict the export to a subset of documents and their entities using the 'File types' and 'Project directory' filters.

DB import

Docker

• identify your neo4j instance container ID:

  Copy

  docker ps | grep neo4j # Should display your running neo4j container ID

• copy your the graph dump inside your neo4j container import directory:

  Copy

  docker cp \
      <export-path> \
      <neo4j-container-id>:/var/lib/neo4j/imports/datashare-graph.dump

• Copy

  docker exec -it <neo4j-container-id> /bin/bash
  ./bin/cypher-shell -f imports/datashare-graph.dump 

Neo4j Desktop import

• open 'Cypher shell':

• copy your the graph dump inside your neo4j instance import directory:

  Copy

  cp <export-path> imports

• Copy

  ./bin/cypher-shell -f imports/datashare-graph.dump 

You will now be able to explore the graph imported in your own neo4j instance.

Explore and visualize entity links

Connect to your database

Visualize and explore with Neo4j Bloom

Neo4j Bloom is accessible from inside Neo4j Desktop app.

Find out more information about to use Neo4j Bloom to explore your graph with:

Query the graph with Neo4j Browser

The Neo4j Browser is available for both Enterprise and Community distributions. You can access it:

Visualize and explore with Linkurious Enterprise Explorer

Find out more information about Linkurious:

Visualize with Gephi

Find out more information about:

Export your graph in the GraphML format

In case you want to restrict the size of the exported graph, you can restrict the export to a subset of documents and their entities using the 'File types' and 'Project directory' filters.

Warning: this requires some technological knowledge.

You can make Datashare follow soft links : add --followSymlinks when Datashare is launched.

If you're on Mac or Windows, you must change the launch script.

If you're on Linux, you can add the option after the Datashare command.

Separate Processing Stages

Execute the SCAN and INDEX stages independently to optimize resource allocation and efficiency.

Examples:

datashare --mode CLI --stage SCAN --redisAddress redis://redis:6379 --busType REDIS
datashare --mode CLI --stage INDEX --redisAddress redis://redis:6379 --busType REDIS

Distribute the INDEX Stage

Leverage Parallelism

Datashare offer --parallelism and --parserParallelism options to enhance processing speed.

Example (for g4dn.8xlarge with 32 CPUs):

datashare --mode CLI --stage INDEX --parallelism 14 --parserParallelism 14
datashare --mode CLI --stage NLP --parallelism 14 --nlpParallelism 14

Optimize ElasticSearch

ElasticSearch can significantly consume CPU and memory, potentially becoming a bottleneck. For production instance of Datashare, we recommend deploying ElasticSearch on a remote server to improve performances.

Adjust JAVA_OPTS

You can fine-tune the JAVA_OPTS environment variable based on your system's configuration to optimize Java Virtual Machine memory usage. Example (for g4dn.8xlarge8with 120 GB Memory):

JAVA_OPTS="-Xms10g -Xmx50g" datashare --mode CLI --stage INDEX

Specify Document Language

If the document language is known, explicitly setting it can save processing time.

• Use --language for general language setting (e.g., FRENCH, ENGLISH).

• Use --ocrLanguage for OCR tasks to specify the Tesseract model (e.g., fra, eng).

Example:

datashare --mode CLI --stage INDEX --language FRENCH --ocrLanguage fra
datashare --mode CLI --stage INDEX --language CHINESE --ocrLanguage chi_sim
datashare --mode CLI --stage INDEX --language GREEK --ocrLanguage ell

Manage OCR Tasks Wisely

OCR tasks are resource-intensive. If not needed, disabling OCR can significantly improve processing speed. You can disable OCR with --ocr false.

Example:

datashare --mode CLI --stage INDEX --ocr false

Efficient Handling of Large Files

Large PST files or archives can hinder processing efficiency. We recommend extract these files before processing with Datashare. If they are too many of them, keep in mind Datashare will be able to extract them anyway.

readpst -reD <Filename>.pst

Star a document

• Hover the document's title in the main column. A button with the labe 'Star document' appears.

• Click it:

The starred document is now marked with a star icon:

Unstar a document

Click again on its star icon. The document won't have a star anymore.

Find your starred documents

• Open the 'Starred' filter on the left:

• Tick 'Starred' (respectively 'Unstarred').

• Only starred (respectively unstarred) documents will be displayed in the main column:

Datashare was created with scalability in mind which gave ICIJ the ability to index terabytes of documents.

To do so, we used a cluster of dozens of EC2 instances on AWS, running on Ubuntu 16.04 and 18.04. We used c4.8xlarge instances (36 CPUs / 60 GB RAM).

You can use Datashare with multiple users accessing a centralized database on a server.

Warning: to put the server mode in place and to maintain it requires some technical knowledge.

Yes, you can remove documents from Datashare. But at the moment, it will remove all your documents. You cannot remove only some documents.

Click the pink trash icon on the bottom left of Datashare:

And then click 'Yes':

You can them re-analyze a new corpus.

For advanced users only - if you'd like to do it with the Terminal, here are the instructions:

• If you're using Mac: rm -Rf ~/Library/Datashare/index

• If you're using Windows: rd /s /q "%APPDATA%"\Datashare\index

• If you're using Linux: rm -Rf ~/.local/share/datashare/index

If you search "Shakespeare" in the search bar and if you run a query containing "Shakespeare" in a batch search, you can get slightly different documents between the two results.

Why?

For technical reasons, Datashare processes both queries in 2 different ways:

a. Search bar (a simple search processed in the browser):

The search query sent to Elasticsearch is processed in your browser by Datashare's client. It is then sent to Elasticsearch through Datashare server which forwards your query.

b. Batch search (several searches processed by the server):

1. Datashare's server processes each of the batch search's queries

2. Each query is sent to Elasticsearch. The results are saved into a database

3. When the batch search is finished, you get the results from Datashare

4. Datashare sends back the results stored into the database/

Datashare's team attempts to make both results be similar, but slight differences can happen between the two queries.

Tarentula is a tool made for advanced users to run bulk actions in Datashare, like:

A named entity in Datashare is the name of an individual, an organization or a location.

Datashare’s Named Entity Recognition (NER) uses pipelines of Natural Language Processing (NLP), a branch of artificial intelligence, to automatically highlight named entities in your documents.

Mac