In local mode, Datashare provides a self-contained software application that users can install and run on their own local machines. The software allows users to search into their documents within their own local environments, without relying on external servers or cloud infrastructure. This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.

What is Datashare?

Welcome to Datashare - a self-hosted documents search software. It is a free and open-source software developed by the International Consortium of Investigative Journalists (ICIJ). Initially created to combine multiple named-entity recognition pipelines, this tool is now a fully-featured search interface to dig into your documents. With the help of several open source tools (Extract, Apache Tika, Apache Tesseract, CoreNLP, OpenNLP, Elasticsearch, etc), Datashare can be used on one single personal computer as well as on 100 interconnected servers.

Who uses it?

Datashare is developed by the ICIJ, a collective of investigative journalists. Datashare is built at the top of technologies and methods already tested with investigations like the Panama Papers or the Luanda Leaks. Seeing the growing interest for ICIJ's technology, we decided to open source this key component of our investigations so a single journalist as well as big media organizations could use it on their own documents.

Curious to know more about how we use Datashare?

• How ICIJ analysed 715,000 Luanda Leaks records

• Help test and improve our latest journalism tool

• How Datashare project will help journalists breach borders

Where can I see it in action?

We setup a Demo instance of Datashare with a small set of documents from the Luxleaks investigation (2014). When using this instance, you will be assigned a temporary user which can star, tag, recommend and explore documents.

Can I run it on my server?

Datashare was also built to run on a server. This is how we use it for our collaborative projects. Please refer to the server documentation to know how it works.

Can I customize Datashare?

When building Datashare, one of our first decisions was to use Elasticsearch to create an index of documents. It would be fair to describe Datashare as a nice looking web interface for Elasticsearch. We want our search platform to be user-friendly while keeping all the powerful Elasticsearch features available for advanced users. This way we ensure that Datashare is usable by non tech-savvy reporters, but still robust enough to satisfy data analysts and developers who want to query the index directly with our API.

We implemented the possibility to create plugins, to make this process more accessible. Instead of modifying Datashare directly, you could isolate your code with a specific set of features and then configure Datashare to use it. Each Datashare user could pick the plugins they need or want, and have a fully customized installation of our search platform. Please have a look at the documentation.

Translations

This project is currently available in English, French, Spanish and Japanese. You can help us to improve and complete translations on Crowdin.

To report a bug, please contribute detailing your logs with:

• your Operating System (Mac, Windows or Linux)

• the version of your Operating System

• the version of Datashare

• screenshots of your issue

• a description of your issue.

If for confidentiality reasons you don't want to open an issue on Github, please write to and our team will do its best to answer you in a timely manner.

Once Datashare is installed, go to "Finder", then "Applications", and double-click on "Datashare".

A Terminal window called 'Datashare.command' opens and describes the technical operations going on during the opening.

Keep this Terminal window open as long as you use Datashare.

Datashare should now automatically open in your default internet browser. If it doesn’t, type "localhost:8080" in your browser. Datashare must be accessed from your internet browser (Firefox, Chome, etc), even though it works offline without Internet connection (see FAQ: Can I use Datashare with no internet connection?).

You can now add documents to Datashare.

This guide will explain to you how to install Datashare on Mac. The installer will take care of checking your system have all the dependencies to run Datashare. Because this software use Apache Tesseract (to perform Optical Character Recognition) and Mac doesn't support them out-of-box, heavy dependencies must be downloaded. If your system have none of those dependencies, the first installation of Datashare can take up to 30 minutes.

The installer will setup:

• Xcode Command Line Tools (if neither XCode or Xcode Command Line Tools are installed)

• MacPorts (if neither Homebrew or MacPorts are installed)

• Apache Tesseract with MacPorts or Homenrew

• Java JRE 17

• Datashare executable

Note: previous versions of this document refered to a "Docker Installer". We do not provide this installer anymore but Datashare is still published on the Docker Hub and supported with Docker.

1. Download Datashare

Go to datashare.icij.org, scroll down and click 'Download for Mac'.

2. Start the installer

Go to your "Downloads" directory in Finder and double-click "datashare-X.Y.Z.pkg":

3. Go through the Datashare Installer

Click 'Continue', 'Install', enter your password and 'Install Software':

The installation begins. You see a progress bar. It stays a long time on "Running package scripts" because it is installing XCode Command Line Tools, MacPort, Tesseract OCR, Java Runtime Environment and finally Datashare.

You can see what it actually does by typing command+L, it will open a window which logs every action made.

In the end, you should see this screen:

You can now start Datashare !

Open the Windows main menu at the left of the bar at the bottom of your computer screen and click on 'Datashare'. (The numbers after 'Datashare' just indicate which version of Datashare you installed.)

A window called 'Terminal' will have opened, showing the progress of opening Datashare. Do not close this black window as long as you use Datashare.

Keep this Terminal window open as long as you use Datashare.

Datashare should now automatically open in your default internet browser. If it doesn’t, type "localhost:8080" in your browser. Datashare must be accessed from your internet browser (Firefox, Chome, etc), even though it works offline without Internet connection (see FAQ: Can I use Datashare with no internet connection?).

It's now time to add documents to Datashare.

Web modes

Those two modes are the only one who create

In local mode and embedded mode, Datashare provides a self-contained software application that users can install and run on their own local machines. The software allows users to search into their documents within their own local environments, without relying on external servers or cloud infrastructure. This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.

In server mode, Datashare operates as a centralized server-based system. Users can access to the platform through a web interface, and the documents are stored and processed on Datashare's servers. This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.

Comparaison between modes

The running modes offer advantages and limitations. This matrix summarizes the differences:

When running Datashare in local mode, users can choose to use embedded services (like ElasticSearch, SQLITE, in-memory key/value store) on the same JVM than Datashare. This variant of the local mode is called "embedded mode" and allows user to run Datashare without having to setup any additional software. The embedded mode is used by default.

CLI mode

In cli mode, Datashare starts without a web server and allow user to perform task over their documents. This mode can be used in conjunction both with local and server modes, while allowing users to distribute heaving task between several servers.

If you want to learn more about which tasks you can execute in this mode, checkout the stages documentation.

Daemon modes

Those modes are intended to be used for action that requires to "wait" for pendings tasks.

In batch download mode, the daemon wait for a user to request a batch download of documents. When a request is receive, the daemon start a task to download the document matching the user search, a bundle them into a zip file.

In batch search mode, the daemon wait for a user to request a batch search of documents. To create a batch search, users must go through the dedicated form on Datashare where they can to upload a list of search terms (in CSV format). The daemon will then start a task to search all matching document and store every occurences in the database.

How to change modes

Datashare is shipped as a single executable, with all modes available. As previously mentioned, the default mpode is the embedded mode. Yet when starting Datashare in command line, you can explicitely specify the running mode. For instance on Ubuntu/Debian:

datashare \
  # Switch to SERVER mode
  --mode SERVER \
  # Dummy session filter to creates ephemeral users
  --authFilter org.icij.datashare.session.YesCookieAuthFilter \
  # Name of the default project for every user
  --defaultProject local-datashare \
  # URI of Elasticsearch
  --elasticsearchAddress http://elasticsearch:9200 \
  # URI of Redis 
  --redisAddress redis://redis:6379 \
  # store user sessions in Redis.
  --sessionStoreType REDIS

The CLI stages are primarly intented to be run for an instance of Datashare that uses non-embedded resources (ElasticSearch, database, key/value memory store). This allows you to distribute heaving tasks between servers.

1. SCAN

This is the first step to add documents to Datashare from the command-line. The SCAN stage allows you to queue all the files that need to be indexed (next step). Once this task is done, you can move to the next step. This stage cannot be distributed.

datashare --mode CLI \  
  # Select the SCAN stage
  --stage SCAN \
  # Where the document are located
  --dataDir /path/to/documents \
  # Store the queued files in Redis
  --dataBusType REDIS \
  # URI of Redis 
  --redisAddress redis://redis:6379

2. INDEX

The INDEX stage is probably the most important (and heavy!) one. It pulls documents to index from the queue create in the previous step, then use a combination of Apache Tika and Tesseract to extract text, metadata and OCR images. The result documents are stored in ElasticSearch. The queue used to store documents to index is a "blocking list", meaning that only one client can pull a concurent value at the time. This allows users to distribute this command on serveral servers.

datashare --mode CLI \
  # Select the INDEX stage
  --stage INDEX \
  # Where the document are located
  --dataDir /path/to/documents \
  # Store the queued files in Redis
  --dataBusType REDIS \
  # URI of Elasticsearch
  --elasticsearchAddress http://elasticsearch:9200 \
  # Enable OCR \
  --ocr true
  # URI of Redis 
  --redisAddress redis://redis:6379

3. NLP

Once a document is available for search (stored in ElasticSearch), you can use the NLP stage to extract named entities from the text. This process will not only create named entities mentions in ElasticSearch, it will mark every analyzed document with the corresponding NLP pipeline (CORENLP by default). In other words, the process is idempotent and can be paralelized as well on several servers.

datashare --mode CLI \
  # Select the NLP stage
  --stage NLP \
  # Use CORENLP to detect named entities
  --nlpp CORENLP \
  # URI of Elasticsearch
  --elasticsearchAddress http://elasticsearch:9200 

Prerequisites

Datashare platform is designed to function effectively by utilizing a combination of various services. To streamline the development and deployment workflows, Datashare relies on the use of Docker containers. Docker provides a lightweight and efficient way to package and distribute software applications, making it easier to manage dependencies and ensure consistency across different environments.

Read more about how to install Docker on your system.

Starting Datashare with a single container

To start Datashare within a Docker container, you can use this command:

docker run --mount src=$HOME/Datashare,target=/home/datashare/data,type=bind -p 8080:8080 icij/datashare:11.1.9 --mode EMBEDDED

Make sure the Datashare folder exists in your homedir or this command will fail. This is an example about how to use Datashare with Docker, data will not be persisted.

Starting Datashare with multiple containers

Within Datashare, Docker Compose can play a significant role in enabling the setup of separated and persistent services for essential components such as the database (PostgreSQL), the search index (Elasticsearch), and the key-value store (Redis).

By utilizing Docker Compose, you can define and manage multiple containers as part of a unified service. This allows for seamless orchestration and deployment of interconnected services, each serving a specific purpose within the Datashare ecosystem.

Specifically, Docker Compose allows you to configure and launch separate containers for PostgreSQL, Elasticsearch, and Redis. These containers can be set up in a way that ensures their data is persistently stored, meaning that any information or changes made to the database, search index, or key-value store will be retained even if the containers are restarted or redeployed.

This separation of services using Docker Compose provides several advantages. It enhances modularity, scalability, and maintainability within the Datashare platform. It allows for independent management and scaling of each service, facilitating efficient resource utilization and enabling seamless upgrades or replacements of individual components as needed.

To start Datashare with Docker Compose, you can use the following docker-compose.yml file:

version: "3.7"
services:

  datashare:
    image: icij/datashare:18.1.3
    hostname: datashare
    ports:
      - 8080:8080
    environment:
      - DS_DOCKER_MOUNTED_DATA_DIR=/home/datashare/data
    volumes:
      - type: bind
        source: ${HOME}/Datashare
        target: /home/datashare/data
      - type: volume
        source: datashare-models
        target: /home/datashare/dist
    command: >-
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password 
      --mode LOCAL
      --tcpListenPort 8080
    depends_on:
      - postgresql
      - redis
      - elasticsearch

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
    restart: on-failure
    volumes:
      - type: volume
        source: elasticsearch-data
        target: /usr/share/elasticsearch/data
        read_only: false
    environment:
      - "http.host=0.0.0.0"
      - "transport.host=0.0.0.0"
      - "cluster.name=datashare"
      - "discovery.type=single-node"
      - "discovery.zen.minimum_master_nodes=1"
      - "xpack.license.self_generated.type=basic"
      - "http.cors.enabled=true"
      - "http.cors.allow-origin=*"
      - "http.cors.allow-methods=OPTIONS, HEAD, GET, POST, PUT, DELETE"

  redis:
    image: redis:4.0.1-alpine
    restart: on-failure

  postgresql:
    image: postgres:12-alpine
    environment:
      - POSTGRES_USER=datashare
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=datashare
    volumes:
      - type: volume
        source: postgresql-data
        target: /var/lib/postgresql/data

volumes:
  datashare-models:
  elasticsearch-data:
  postgresql-data:

Open a terminal or command prompt and navigate to the directory where you saved the docker-compose.yml file. Then run the following command to start the Datashare service:

docker-compose up -d

The -d flag runs the containers in detached mode, allowing them to run in the background.

Docker Compose will pull the necessary Docker images (if not already present) and start the containers defined in the YAML file. Datashare will take a few seconds to start. You can check the progression of this opperation with:

docker-compose logs -f datashare

Once the containers are up and running, you can access the Datashare service by opening a web browser and entering http://localhost:8080. This assumes that the default port mapping of 8080:8080 is used for the Datashare container in the YAML file.

That's it! You should now have the Datashare service up and running, accessible through your web browser. Remember that the containers will continue to run until you explicitly stop them.

To stop the Datashare service and remove the containers, you can run the following command in the same directory where the docker-compose.yml file is located:

docker-compose down

This will stop and remove the containers, freeing up system resources.

Standard (Windows 7 Service Pack 2 or newer version)

Uninstall any prior standard version

Before we start, please uninstall any prior standard version of Datashare if you had already installed it. You can follow these steps: https://www.laptopmag.com/articles/uninstall-programs-windows-10

Download Datashare

Go to datashare.icij.org, scroll down and click 'Download for free'.

The file "datashare-X.Y.Z.exe" is now downloaded. Double click on the name of the file in order to execute it.

Allow Datashare to install

As Datashare is not signed, this popup asks for your permission. Don't click 'Don't run' but click 'More info':

Click 'Run anyway':

It asks if you want to allow the app to make changes to your device. Click 'Yes':

Install Datashare

On the Installer Wizard, as you need to download and install OpenJDK11 if it is not installed on your device, click 'Install':

The following windows with progress bars will be displayed:

Choose a language and click 'OK':

Install Tesseract OCR

To install Tesseract OCR, click the following buttons on the Installer Wizard's windows:

Untick 'Show README' and click 'Finish':

Finally, click "Close" to close the installer of TesseractOCR.

Install Datashare.jar

It now downloads the back end and the front end, Datashare.jar:

When it is finished, click 'Close':

You can now start Datashare!

Find your Datashare folder on your Mac

Open your Mac's 'Finder' by clicking on the blue smiling icon in your Mac's 'Dock':

On the menu bar at the top of your computer, click 'Go'. Click on 'Home' (the house icon).

You will see a folder called 'Datashare':

If you want to quickly access it in the future, you can drag and drop it in 'Favorites' on the left of this window:

Add documents in your Datashare folder

Copy or place the documents you want to have in Datashare in this Datashare folder.

Launch Datashare

Open your Applications. You should see Datashare. Double click on it:

Datashare opens in your default internet browser. Click 'Tasks':

Click the 3rd tab 'Analyze your documents':

You can now analyze your documents in Datashare.

Start Datashare by launching it from the command-line:

datashare

Datashare should now automatically open in your default internet browser. If it doesn’t, type "localhost:8080" in your browser. Datashare must be accessed from your internet browser (Firefox, Chome, etc), even though it works offline without Internet connection (see: Can I use Datashare with no internet connection?).

It's now time to add documents to Datashare.

Currently, only a .deb package for Debian/Ubuntu is provided.

If you want to run it with another Linux distribution, you can download the latest version of the Datashare jar here : https://github.com/ICIJ/datashare/releases/latest

And adapt the following launch script : https://github.com/ICIJ/datashare/blob/master/datashare-dist/src/main/deb/bin/datashare to your environment.

1. Download Datashare

Go to datashare.icij.org, scroll down and click 'Download .deb'

Save the Debian package as a file

2. Install the package

$ sudo apt install /dir/to/debian/package/datashare-dist_7.2.0_all.deb

3. Run Datashare with:

$ datashare

You can now start Datashare!

Add documents

1. To add your documents in Datashare, click 'Tasks' in the left menu:

2. Click 'Analyze your documents':

3. Click 'Add documents' so Datashare can extract the texts from your files:

Options when adding documents

You can:

• Select the specific folder or sub-folder containing the documents you want to add.

• Extract text also from images/PDFs (OCR). Be aware the indexing can be up to 10 times longer.

• Skip already indexed files.

Two extraction tasks are now running: the first is the scanning of your Datashare folder which sees if there are new documents to analyze (ScanTask). The second is the indexing of these files (IndexTask):

It is not possible to 'Find people, organizations and locations' while of these two tasks is still running.

When tasks are done, you can start exploring documents by clicking 'Search' in the left menu but you won't have the named entities (names of people, organizations and locations) yet. To have these, follow the steps below.

Extract names of people, organizations and locations

1. After the text is extracted, you can launch named entities recognition by clicking the button 'Find people, organizations and locations'.

2. In the window below, you are asked to choose between finding Named Entities or finding email addresses (you cannot do both simultaneously, you need to do one after the other, no matter the order):

You can now see running tasks and their progress. After they are done, you can click 'Clear done tasks' to stop displaying tasks that are completed.

3. You can search your indexed documents without having to wait for all tasks to be done. To access your documents, click 'Search':

Extract email addresses

To extract email addresses in your documents:

• Re-click on 'Find people, organizations, locations and email addresses' (in Tasks (left menu) > Analyze your documents)

• Click the second radio button 'Find email addresses':

Plugins are small programs that you can add to Datashare's front-end to get new features (the front-end is the interface, "the part of the software with which the user interacts directly" - ).

Extensions are small programs that you can add to Datashare's back-end to add new features (the back-end is "the part of the software that is not directly accessed by the user, typically responsible for storing and manipulating data" - ).

Add plugins to Datashare (front-end)

1. Go to "Settings":

1. Click "Plugins":

1. Choose the plugin you want to add and click "Install now":

If you want to install a plugin from an URL, click "Install plugin from URL".

1. Your plugin is installed.

Refresh your page to see your new plugin activated in Datashare.

Add extensions to Datashare (back-end)

1. Go to "Settings":

1. Click "Extensions":

1. Choose the extension you want to add and click "Install now":

If you want to install an extension from an URL, click "Install extension from URL".

1. Your extension is installed.

Restart Datashare to see your new extension activated in Datashare.

Update plugin or extension with latest version

When a newer version of a plugin or extension is available, you can click on the "Update" button to get the latest version.

After that, if it is a plugin, refresh your page to activate the latest version.

If it is an extension, restart Datashare to activate the latest version.

Create your own plugin or extension

People who code can create their own plugins and extensions by following these steps:

Install the neo4j plugin

Install the neo4j plugin following instructions available in the .

Configure the neo4j plugin

1. Go to "Settings":

2. Make sure the following settings are properly set:

• Neo4j Host should be localhost or the address where your neo4j instance is running

• Neo4j Port should be the port where your neo4j instance is running (7687 by default)

• Neo4j User should be set to your neo4j user name (neo4j by default)

• Neo4j Password should only be set if your neo4j user is using password authentication

3. When running Neo4j Community Edition, set the Neo4j Single Project value. In community edition, the neo4j DBMS is restricted to a single database. Since Datashare supports multiple projects, you must set the Neo4j Single Project with the name of the project which will use neo4j plugin. Other projects won't be able to use the neo4j plugin.

4. Restart Datashare to apply the changes

5. You should be able to see the neo4j widget in your project page, after a little while its status should be RUNNING:

Next step

Prerequisites

Get neo4j up and running

Follow the instructions of the to get neo4j up and running.

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the 'Other platforms and versions' button when downloading to access versions if necessary.

Add entities

If it's not done yet and extract both names of people, organizations and locations as well as email addresses.

If your project contains email documents, make sure to also extract email addresses.

Next step

You can now !

To be able to perform OCR, Datashare uses an open source technology called Apache Tesseract. When Tesseract extracts text from images, it uses "language packages" especially trained for each specific languages. Unfortunately, those packages can be heavy and to ensure a lightweight installation of Datashare, the installer doesn't use them all by default. In the case Datashare informs you of a missing package, this guide will explain you how to manually install it on your system.

Install packages on Linux

To add ocr languages on linux, simply use the following command:

Where `[lang]` is can be :

• all if you want to install all languages

• a language code (ex: fra, for French), the list of languages is available

Install packages on Mac

The Datashare Installer for Mac checks for the existence of either or , which packages managers are used for the installation of Tesseract. If none of those two package managers is present, the Datashare Installer will install MacPorts by default.

With MacPorts (default)

First, you must check that MacPort is installed on your computer. Please run in a Terminal:

You should see an output similar to this:

If MacPort is installed on your computer, you should be able to add the missing Tesseract language package with the following command (for German):

Once the installation is done, simply close and restart Datashare to be able to use the newly installed packages.

With Homebrew

If Homebrew was already present on your system when Datashare was installed, Datashare used it to install Tesseract and its language packages. Because Homebrew doesn't package each Tesseract language individually, all languages are already supported by your system. In other words, you have nothing to do!

If you want to check if Homebrew is installed, run the following command in a Terminal:

You should see an output similar to this:

Install languages on Windows

*Additionnal languages can be also added during Tesseract's installation.

The list of installed languages can be checked with Windows command prompt or Powershell with the commandtesseract --list-langs.

Datashare has to be restarted after the language installation.

Authentication with Datashare in server mode is the most impacting choice that has to be made. It can be one of the followings:

• basic authentication with credentials stored in database (PostgreSQL)

• basic authentication with credentials stored in Redis

• OAuth2 with credentials provided by an identity provider (KeyCloak for example)

• dummy basic auth to accept any user (⚠️ if the service is exposed to internet, it will leak your documents)

Create the graph

1. Open the 'Projects' page and select your project:

1. Create the graph by clicking on the 'Create graph' button inside the neo4j widget:

You will see a new import task running:

When the graph creation is complete, 'Graph statistics' will reflect the number of documents and entities nodes found in the graph:

Update the graph

When new documents or entities are added or modified inside Datashare, you will need to update the neo4j graph to reflect these changes.

To update the graph click on the 'Update graph' button inside the neo4j widget:

To detect whether a graph update is needed you can compare the number of documents found inside Datashare to the number found in the 'Graph statistics' and run an update in case of mismatch:

The update will always add missing nodes and relationships, update existing ones if they were modified, but will never delete graph nodes or relationships.

Next steps

• explore your graph using your favorite visualization tool

Prerequisites

Datashare platform is designed to function effectively by utilizing a combination of various services. To streamline the development and deployment workflows, Datashare relies on the use of Docker containers. Docker provides a lightweight and efficient way to package and distribute software applications, making it easier to manage dependencies and ensure consistency across different environments.

Read more about how to install Docker on your system.

Starting Datashare with multiple containers

Within Datashare, Docker Compose can play a significant role in enabling the setup of separated and persistent services for essential components. By utilizing Docker Compose, you can define and manage multiple containers as part of a unified service. This allows for seamless orchestration and deployment of interconnected services, each serving a specific purpose within the Datashare ecosystem.

Specifically, Docker Compose allows you to configure and launch separate containers for PostgreSQL, Elasticsearch, and Redis. These containers can be set up in a way that ensures their data is persistently stored, meaning that any information or changes made to the database, search index, or key-value store will be retained even if the containers are restarted or redeployed.

This separation of services using Docker Compose provides several advantages. It enhances modularity, scalability, and maintainability within the Datashare platform. It allows for independent management and scaling of each service, facilitating efficient resource utilization and enabling seamless upgrades or replacements of individual components as needed.

To start Datashare in server mode with Docker Compose, you can use the following docker-compose.yml file:

version: "3.7"
services:

  # This is the main Datashare service that serves the web interface. 
  # Here it is configured with a "dummy" authentication backend which 
  # creates epehemeral user sessions.
  datashare_web:
    image: icij/datashare:18.1.3
    hostname: datashare
    ports:
      - 8080:8080
    environment:
      - DS_DOCKER_MOUNTED_DATA_DIR=${HOME}/Datashare
    volumes:
      - type: bind
        source: ${HOME}/Datashare
        target: /home/datashare/Datashare
    depends_on:
      postgresql:
        condition: service_healthy
      redis:
        condition: service_healthy
      elasticsearch:
        condition: service_healthy
    command: >-
      --mode SERVER
      --dataDir /home/datashare/Datashare
      --pluginsDir /home/datashare/plugins
      --extensionsDir /home/datashare/extensions
      --authFilter org.icij.datashare.session.YesCookieAuthFilter
      --busType REDIS
      --batchQueueType REDIS
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password 
      --defaultProject secret-project
      --elasticsearchAddress http://elasticsearch:9200      
      --messageBusAddress redis://redis:6379
      --queueType REDIS
      --redisAddress redis://redis:6379  
      --rootHost http://localhost:8080
      --sessionStoreType REDIS
      --sessionTtlSeconds 43200
      --tcpListenPort 8080

  # We use a service to create the "secret-project". In theory you only need 
  # to run it the first time you start Datashare. 
  datashare_create_project:
    image: icij/datashare:18.1.3
    restart: no
    depends_on:
      elasticsearch:
        condition: service_healthy
    command: >-
      --defaultProject secret-project 
      --mode CLI 
      --stage INDEX 
      --elasticsearchAddress http://elasticsearch:9200

  # This service starts a deamon that wait for background tasks
  # so it can run them (and save them in the database).
  datashare_task:
    image: icij/datashare:18.1.3
    depends_on:
      - datashare_web
    command: >-
      --mode TASK_RUNNER
      --batchQueueType REDIS
      --batchThrottleMilliseconds 500
      --busType REDIS
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password
      --defaultProject secret-project 
      --elasticsearchAddress http://elasticsearch:9200  
      --queueType REDIS
      --redisAddress redis://redis:6379
      --scrollSize 100  
      
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
    restart: on-failure
    volumes:
      - type: volume
        source: elasticsearch-data
        target: /usr/share/elasticsearch/data
        read_only: false
    environment:
      - "http.host=0.0.0.0"
      - "transport.host=0.0.0.0"
      - "cluster.name=datashare"
      - "discovery.type=single-node"
      - "discovery.zen.minimum_master_nodes=1"
      - "xpack.license.self_generated.type=basic"
      - "http.cors.enabled=true"
      - "http.cors.allow-origin=*"
      - "http.cors.allow-methods=OPTIONS, HEAD, GET, POST, PUT, DELETE"
    healthcheck:
      test: ["CMD-SHELL", "curl --silent --fail elasticsearch:9200/_cluster/health || exit 1"]

  postgresql:
    image: postgres:12-alpine
    environment:
      - POSTGRES_USER=datashare
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=datashare
      # This is needed by the heathcheck command
      # @see https://stackoverflow.com/a/60194261
      - PGUSER=datashare
    volumes:
      - type: volume
        source: postgresql-data
        target: /var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready", "-U", "datashare", "-d", "datashare"]
  
  redis:
    image: redis:4.0.1-alpine
    restart: on-failure
    volumes:
      - type: volume
        source: redis-data
        target: /data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli", "--raw", "incr", "ping"]

volumes:
  datashare-batchdownload-dir:
  elasticsearch-data:
  postgresql-data:
  redis-data:

Open a terminal or command prompt and navigate to the directory where you saved the docker-compose.yml file. Then run the following command to start the Datashare service:

docker-compose up -d

The -d flag runs the containers in detached mode, allowing them to run in the background.

Docker Compose will pull the necessary Docker images (if not already present) and start the containers defined in the YAML file. Datashare will take a few seconds to start. You can check the progression of this opperation with:

docker-compose logs -f datashare_web

Once the containers are up and running, you can access the Datashare service by opening a web browser and entering http://localhost:8080. This assumes that the default port mapping of 8080:8080 is used for the Datashare container in the YAML file.

To stop the Datashare service and remove the containers, you can run the following command in the same directory where the docker-compose.yml file is located:

docker-compose down

This will stop and remove the containers, freeing up system resources.

Add documents to Datashare

If you reach that point, Datashare is up and running but you will discover very quickly that no documents is available in the search results. Next step: Add documents from the CLI.

Extract named entities

Datashare as the ability to detect email addresses, name of people, organizations and locations. You must perform the named entities extraction in the same fashion than the previous commands. Final step: Add named entities from the CLI.

In server mode, Datashare operates as a centralized server-based system. Users can access to the platform through a web interface, and the documents are stored and processed on Datashare's servers. This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.

Launch configuration

Datashare is launched with --mode SERVER and you have to provide:

• the external elasticsearch index address elasticsearchAddress

• a Redis store address redisAddress

• a Redis data bus address messageBusAddress

• a database JDBC URL dataSourceUrl

• the host of Datashare (used to generate batch search results URLs) rootHost

• an authentication mechanism and its parameters

Example:

docker run -ti ICIJ/datashare:version --mode SERVER \
    --redisAddress redis://my.redis-server.org:6379 \
    --elasticsearchAddress https://my.elastic-server.org:9200 \
    --messageBusAddress my.redis-server.org \
    --dataSourceUrl jdbc:postgresql://db-server/ds-database?user=ds-user&password=ds-password \
    --rootHost https://my.datashare-server.org
    # ... +auth parameters (see authentication providers section)

This document assumes you have installed Datashare in server mode within Docker and already added documents to Datasharte.

In server mode, it's important to understand that Datashare does not provide an interface to add documents. As there is no build-in roles and permission in Datashare's data model, we have no way to differentiate user to offer admin additional tools.

This is likely to be changed in the near future, but in the meantime, you can extract named entities using the command-line interface.

Datashare as the ability to detect email addresses, name of people, organizations and locations. This process use a Natural Language Processing pipeline called CORENLP. Once your documents have been indexed in Datashare, you can perform the named entities extraction in the same fashion as the previous CLI's stages:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage NLP \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --nlpParallelism 2 \
  --nlpp CORENLP

What's happening here:

• Datashare starts in "CLI" mode

• We ask to process the NLP stage

• We tell Datashare to use the elasticsearch service

• Datashare will pull documents from ElasticSearch directly

• Up to 2 documents will be analyzed in parallel

• Datashare will use the CORENLP pipeline

Datashare will use the output queue from the previous INDEX stage (by default extract:queue:nlp in Redis) that contains all the document ids to be analyzed.

The first time you run this command you will have to wait a little bit because Datashare need to download CORENLP's models which can be big.

You can also use chain the 3 stages altogether:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage SCAN,INDEX,NLP \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --nlpParallelism 2 \
  --nlpp CORENLP \
  --dataDir /home/datashare/Datashare/

As for the previous stages you may want to restore the output queue from the INDEX stage. You can do:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage ENQUEUEIDX,NLP \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --nlpParallelism 2 \
  --nlpp CORENLP

The added ENQUEUEIDX stage will read Elasticsearch index, find all documents that have not already been analyzed by the CORENLP NER pipeline, and put the ids of those documents into the extract:queue:nlp queue.

This document assumes you have installed Datashare in server mode within Docker.

In server mode, it's important to understand that Datashare does not provide an interface to add documents. As there is no build-in roles and permission in Datashare's data model, we have no way to differenciate user to offer admin additional tools.

This is likelly to be changed in the near future, but in the meantime, you can still add documents to Datashare using the command-line interface.

Here is a simple command to scan a directory and index its files:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage SCAN,INDEX \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --dataDir /home/datashare/Datashare/

What's happening here:

• Datashare starts in "CLI" mode

• We ask to process both SCAN and INDEX stages at the same time

• The SCAN stage feeds a queue in memory with file to add

• The INDEX stage pulls files from the queue to add them to ElasticSearch

• We tell Datashare to use the elasticsearch service

• Files to add are located in /home/datashare/Datashare/ which is a directory mounted from the host machine

Alternativly, you can do this in two separated phases, as long as you tell Datashare to store the queue in a shared resource. Here, we use the redis:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage SCAN \
  --queueType REDIS \
  --queueName "datashare:queue" \
  --redisAddress redis://redis:6379 \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --dataDir /home/datashare/Datashare/

Once the opperation is done, we can easily check the content of queue created by Datashare in redis. In this example we only display the 20 first files in the datashare:queue:

docker compose exec redis redis-cli lrange datashare:queue 0 20

The INDEX stage can now be executed in the same container:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage INDEX \
  --queueType REDIS \
  --queueName "datashare:queue" \
  --redisAddress redis://redis:6379 \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --dataDir /home/datashare/Datashare/

Once the indexing is done, Datashare will exit gracefully and your document will already be visible on Datashare.

Sometimes you will face the case where you have an existing index, and you want to index additional documents inside your working directory without processing every document again. It can be done in two steps :

• Scan the existing ElasticSearch index and gather document paths to store it inside a report queue

• Scan and index (with OCR) the documents in the directory, thanks to the previous report queue, it will skip the paths inside of it

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage SCANIDX \
  --queueType REDIS \
  --reportName "report:queue" \
  --redisAddress redis://redis:6379 \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --dataDir /home/datashare/Datashare/

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage SCAN,INDEX \
  --ocr true \
  --queueType REDIS \
  --queueName "datashare:queue" \
  --reportName "report:queue" \
  --redisAddress redis://redis:6379 \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --dataDir /home/datashare/Datashare/

Basic authentication is a simple protocol that uses the HTTP headers and the browser to authenticate users. User credentials are sent to the server in the header Authorization with user:password base64 encoded:

Authorization: Basic dXNlcjpwYXNzd29yZA==

It is secure as long as the communication to the server is encrypted (with SSL for example).

On the server side, you have to provide a user store for Datashare. For now we are using a Redis data store.

So you have to provision users. The passwords are sha256 hex encoded. For example using bash:

$ echo -n bar | sha256sum
fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9  -

Then insert the user like this in Redis:

$ redis-cli -h my.redis-server.org
redis-server.org:6379> set foo '{"uid":"foo", "password":"fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9", "groups_by_applications":{"datashare":["local-datashare"]}}'

If you use other indices, you'll have to include them in the group_by_applications, but local-datashare should remain. For exammple if you use myindex:

$ redis-cli -h my.redis-server.org
redis-server.org:6379> set foo '{"uid":"foo", "password":"fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9", "groups_by_applications":{"datashare":["myindex","local-datashare"]}}'

Then you should see this popup:

Example

Here is an example of launching Datashare with Docker and the basic auth provider filter backed in Redis:

docker run -ti ICIJ/datashare --mode SERVER \
    --batchQueueType REDIS \
    --dataSourceUrl 'jdbc:postgresql://postgres/datashare?user=<username>&password=<password>' \
    --sessionStoreType REDIS \
    --authFilter org.icij.datashare.session.BasicAuthAdaptorFilter \
    --authUsersProvider org.icij.datashare.session.UsersInRedis

Prerequisites

Get neo4j up and running

Follow the instructions of the to get neo4j up and running.

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the 'Other platforms and versions' button when downloading to access versions if necessary.

Add entities

If it's not done yet add entities to your project .

If your project contains email documents, make sure to run the EMAIL pipeline together with regular NLP pipeline. To do so add set the follow nlpp flag to --nlpp CORENLP,EMAIL.

Next step

You can now !

This is the default authentication mode: if not provided in CLI it will be selected. With OAuth2 you will need a third-party authorization service. The diagram below describes the workflow:

Example

Integration with KeyCloak

Install the neo4j plugin

Install the neo4j plugin using the Datashare CLI so that users can access it from the frontend:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --pluginInstall datashare-plugin-neo4j-graph-widget 

Installing the plugin install the datashare-plugin-neo4j-graph-widget plugin inside /home/datashare/plugings and will also install the datashare-extension-neo4j backend extension inside /home/datashare/extensions. These locations can be changed by updating the docker-compose.yml.

Configure the neo4j extension

Update the docker-compose.yml to reflect your neo4j docker service settings.

...
services:
    datashare_web:
      ...
      environment:
        - DS_DOCKER_NEO4J_HOST=neo4j
        - DS_DOCKER_NEO4J_PORT=7687
        - DS_DOCKER_NEO4J_SINGLE_PROJECT=secret-project  # This is for community edition only

If your choose a different neo4j user or set a password for your neo4j user make sure to also set DS_DOCKER_NEO4J_USER and DS_DOCKER_NEO4J_PASSWORD.

When running Neo4j Community Edition, set the DS_DOCKER_NEO4J_SINGLE_PROJECT value. In community edition, the neo4j DBMS is restricted to a single database. Since Datashare supports multiple projects, you must set the DS_DOCKER_NEO4J_SINGLE_PROJECT with the name of the project which will use neo4j plugin. Other projects won't be able to use the neo4j plugin.

Restart Datasahre

After installing the plugin a restart might be needed for the plugin to display:

docker compose restart datashare_web

Next step

You can now create the graph !

Apply filters

On the left column, you can apply filters by ticking them, like 'Portable Document Format (PDF)' in File Types and 'English' in Languages in the example below:

A reminder of the currently applied filters, as well as your queried terms, are displayed at the top of the results' column. You can easily unselect these filters from there by clicking them or clear all of them:

The currently available filters are:

• Projects: if you have more than one project, you can select several of them and run searches in multiple projects at once.

• Starred: If you have starred documents, you can easily find them again.

• Tags: If you wrote some tags, you will be able to select and search for them.

• Recommended by: available only on server (collaborative) mode, this functionality helps you find the document recommended by you and/or others.

• File type: This is the 'Content type' of the file (Word, PDF, JEPG image, etc.) as you can read it in a document's 'Tags & Details'.

• Creation dates: the calendar allows you to select a single creation date or a date range. This is when the document was created as it is noticed in their properties. You can find this in a document's 'Tags & Details'.

• Languages: Datashare detects the main language of each document.

• People / Organizations / Locations: you can selected these named entities and search them.

• Path: This is the location of your documents as it is indicated in your original files (ex: desktop/importantdocuments/mypictures). You can find this in a document's 'Tags & Details'.

• Indexing date: This date corresponds to when you indexed the documents in Datashare.

• Extraction level: This regards embedded documents. The file on disk is level zero. If a document (pictures, etc) is attached or contained in a file on disk, extraction level is “1st”. If a document is attached or contained in a document itself contained in a file on disk, extraction level is “2nd”, etc.

Filters can be combined together and combined with searches in order to refine results.

Filter by named entities

If you have asked Datashare to 'Find people, organizations and locations', you can see names of individuals, organizations and locations in the filters. These are the named entities automatically detected by Datashare.

Search for named entities in the filter's search bar:

Select all of them, one or several of them to filter the documents that mention them:

Use the 'Exclude' button

If you want to select all items except one or several of them, you can use the 'Exclude button'.

It allows you to search for all documents which do not correspond to the filter(s) you selected, that is to say to the currently strikethrough filters.

Contextualize filters

In several filters, you can tick 'Contextualize' : this will update the number of documents indicated in the filters in order to reflect the results. The filter will only count what you selected.

In the example below, the 'Contextualize' checkboxes are not ticked:

After the Contextualize button in Tags filter is ticked:

After the Languages button in Tags filter is ticked:

Clear all filters

To reset all filters at the same time, click 'Clear all':

Improving the performance of Datashare involves several techniques and configurations to ensure efficient data processing. Extracting text from multiple file types and images is an heavy process so be aware that even if we take care of getting the best performances possible with and , this process can be expensive. Below are some tips to enhance the speed and performance of your Datashare setup.

Separate Processing Stages

Execute the SCAN and INDEX stages independently to optimize resource allocation and efficiency.

Examples:

Distribute the INDEX Stage

Distribute the INDEX stage across multiple servers to handle the workload efficiently. We often use multiple instances (32 CPUs, 128 GB of memory) with a remote Redis and a remote ElasticSearch instance to alleviate processing load.

For projects like the (2.94 TB), we ran the INDEX stage to up to 10 servers at the same time which cost ICIJ several thousand of dollars.

Leverage Parallelism

Datashare offer --parallelism and --parserParallelism options to enhance processing speed.

Example (for g4dn.8xlarge with 32 CPUs):

Optimize ElasticSearch

ElasticSearch can significantly consume CPU and memory, potentially becoming a bottleneck. For production instance of Datashare, we recommend deploying ElasticSearch on a remote server to improve performances.

Adjust JAVA_OPTS

You can fine-tune the JAVA_OPTS environment variable based on your system's configuration to optimize Java Virtual Machine memory usage. Example (for g4dn.8xlarge8with 120 GB Memory):

Specify Document Language

If the document language is known, explicitly setting it can save processing time.

• Use --language for general language setting (e.g., FRENCH, ENGLISH).

• Use --ocrLanguage for OCR tasks to specify the Tesseract model (e.g., fra, eng).

Example:

Manage OCR Tasks Wisely

OCR tasks are resource-intensive. If not needed, disabling OCR can significantly improve processing speed. You can disable OCR with --ocr false.

Example:

Efficient Handling of Large Files

Large PST files or archives can hinder processing efficiency. We recommend extract these files before processing with Datashare. If they are too many of them, keep in mind Datashare will be able to extract them anyway.

You can have a dummy authentication that always accepts basic auth. So you should see this popup:

But then you can enter whatever user or password, it will enter Datashare.

Example

You need an internet connection to install Datashare.

You also need the internet to find people, organizations and locations in documents the first time you use any new NLP option because the models which find these named entities are downloaded when you ask for finding named entities the first time. Subsequently, you don't need an internet connection to find named entities after.

You don't need internet connection:

• to add documents to Datashare

• to find named entities (except for the first time you use an NLP options - see above)

• to search and explore documents

• to download documents

This allows you to work safely on your documents. No third-party should be able to intercept your data and files while you're working offline on your computer.

Search in documents

1. To see all your documents (you need to have added documents to Datashare and have analyzed them before), click 'Search in documents':

If not collapsed yet, to collapse the left menu in order to gain room, click the 'hamburger menu':

2. Search for specific documents. Type terms in the search bar, press Enter or click 'Search':

IMPORTANT:

• To make your searches more precise, you can search with operators (AND, OR, ....): read more here.

• If you get a message "Your search query is wrong", it is probably because you are misusing one or some reserved characters (like ^ " ? ( [ * OR AND etc). Please refer to this page.

3. You can search in specific fields like tags, title, author, recipient, content, path or thread ID. Click 'All fields' and select your choice in the dropdown menu:

Choose between views (list, grid, table)

Select the view on the top right.

• List:

• Grid:

• Table:

Search within a document

Once a document is opened, you can search for terms in this document:

• Press Command (⌘) + F (on Mac) or Control + F (on Windows and Linux) or click on the search bar above your Extracted Text

• Type what you search for

• Press ENTER to go from one occurrence to the next one

• Presse SHIFT + ENTER to go from one occurrence to the previous one

(To know all the shortcuts in Datashare, please read 'Use keyboard shortcuts'.)

This also counts the number of occurrences of your searched terms in this document:

If you run email extraction and searched for one or several email addresses, if the email adresses are in the email's metadata (recipient, sender or other field), there will be a 'in metadata' label attached to the email addresses:

Extracted text

In 'Extracted Text', you can read the text of a document as extracted by Datashare:

Please beware that Datashare show named entities by default. This can overwrite some original text with wrong named entities. It is thus important to always verify original text by deactivating named entity overwriting. To do so, please:

• Turn off the toggle button ‘Show named entities’ and read the extracted text

• Check the ‘Preview’ of original document if available

• Check the original document at its original location or by clicking the pink button ‘Download’

****

Search for attachments

If the documents has attachments (technically called 'children documents'), find them at the end of the document. Click their pink button to open them:

To open all the attachments in Datashare, click 'See children documents' in Tags and Details:

****

Search for terms within this document

• Press Command(⌘) + F (on Mac) or Control + F (on Windows and Linux) or click on the search bar above your Extracted Text

• Type what you search for

• Press ENTER to go from one occurrence to the next one

• Presse SHIFT + ENTER to go from one occurrence to the previous one

(To know all the shortcuts in Datashare, please read 'Use keyboard shortcuts'.)

This also counts the number of occurrences of your searched terms in this document:

If you run email extraction and searched for one or several email addresses, if the email adresses are in the email's metadata (recipient, sender or other field), there will be a 'in metadata' label attached to the email addresses:

Tags & Details

In 'Tags & Details', you can read the document's details. It's all the metadata as they appear in the original file. Please click 'Show more details' to get all metadata:

You can also read the tags you previously wrote for this document, like 'test1', 'test2' and 'test3' in the example below:

You can then search for the documents you tagged:

• Type the tag(s) in the main search bar

• Click 'All fields' and select 'Tags'

• Click 'Search' or press 'Enter'

To learn more about tags, please read 'Tag a document'.

Named Entities

In 'Named Entities', you can read the name of people, organizations and locations as well as the number of their occurrences in the document:

Please beware that there can still be some errors due to the technology of Named Entity Extraction (NER) on which Datashare relies.

If you run email extraction, you will see a list of the extracted emails:

Preview

In 'Preview', you can read the original document.

'Preview' is available for some formats only.

You can sort documents by:

• relevance (by default): it is a score calculated by the search engine

• indexing date: when you analyzed the document, the day and time you 'put' them in Datashare

• creation date: the day and time the document was created, as it is written in the document's metadata

• size of the documents

• path of the documents

You can also decide the number of documents displayed by page (10, 25, 50 or 100):

Recommend a document

Open the document by clicking on its title

• Click the button 'Mark as recommended':

Your recommendation is now displayed on this page and in the left 'Recommended by' filter.

Search recommended documents with the filter

Open the filter untitled 'Recommended by'

Unmark a document as recommended

Open the document and click on "Unmark as recommended".

If you want to search a list of queries in Datashare, instead of doing each of them one by one, you can upload the list directly in Datashare. To do so, you will:

• Create a list of terms that you want to search in the first column of a spreadsheet

• Export the spreadsheet as a CSV (a special format available in any spreadsheet software)

• Upload this CSV in the "new Batch Search" form in Datashare

• Get the results for each query in Datashare - or in a CSV.

Prepare your batch search

Write your queries in a spreadsheet

• Write your queries, one per line and per cell, in the first column of a spreadsheet (Excel, Google Sheets, Numbers, Framacalc, etc.). In the example below, there are 4 queries:

• Do not put line break(s) in any of your cells.

To delete line break(s) in your spreadsheet, you can use the "Find and replace all" functionality. Find all "\n" and replace them all by nothing or a space.

• Write 2 characters minimum in the cells. If one cell contains one character but at least one other cell contains more than one, the cell containing one character will be ignored. If all cells contain only one character, the batch search will lead to 'failure'.

• If you have blank cells in your spreadsheet...

...the CSV (which stand for 'Comma-separated values') will keep these blank cells. It will separate them with semicolons (the 'commas'). You will thus have semicolons in your batch search results (see screenshot below). To avoid that, you need to remove blank cells in your spreadsheet before exporting it as a CSV.

• If there is a comma in one of your cells (like in "1,8 million" in our example above), the CSV will formally put the content of the cell in double quotes in your results and search for the exact phrase in double quotes.

Want to search only on some documents?

In the new Batch Search's form > Advanced Filters, you will be able to select some file types and some paths if you want to search only in some documents.

But you can also use fields directly in your queries in the CSV.

For instance, if you want to search only in some documents with certain tag(s), you can write your queries like this: "Paris AND (tags:London OR tags:Madrid NOT tags:Cotonou)".

Use operators in your CSV

The operators AND NOT * ? ! + - do work in batch searches (as they do in the regular search bar) but only if "Do phrase match" in Advanced filters is turned off.

Reserved characters, when misused, can lead to failures because of syntax errors.

Please also note that searches are not case sensitive: if you search 'HeLlo', it will look for all occurrences of 'Hello', 'hello', 'hEllo', 'heLLo', etc. in the documents.

Export your CSV encoded in UTF-8

Export your spreadsheet in a CSV format like this:

Important: Use the UTF-8 encoding.

• LibreOffice Calc: it uses UTF-8 by default. If not, go to LibreOffice menu > Preferences > Load/Save > HTML Compatibility and make sur the character set is 'Unicode (UTF-8)':

• Microsoft Excel: if it is not set by default, select "CSV UTF-8" as one of the formats, as explained here.

• Google Sheets: it uses UTF-8 by default. Just click "Export to" and "CSV".

• Other spreadsheet softwares: please refer to each software's user guide.

Launch your batch search

• Open Datashare, click 'Batch searches' in the left menu and click 'New batch search' on the top right:

• Type a name for your batch search:

• Upload your CSV:

• Add a description (optional):

• Set the advanced filters ('Do phrase matches', 'Fuzziness' or 'Proximity searches', 'File types' and 'Path') according to your preferences:

What is 'Do phrase matches'?

'Do phrase matches' is the equivalent of double quotes: it looks for documents containing an exact sentence or phrase. If you turn it on, all queries will be search for their exact mention in documents as if Datashare added double quotes around each query.

What is fuzziness?

When you run a batch search, you can set the fuzziness to 0, 1 or 2. It will apply to each term in a query. It corresponds to the maximum number of operations (insertions, deletions, substitutions and transpositions) on characters needed to make one term match the other.

kitten -> sitten (1 substitution (k turned into s) = fuzziness is 1)

kitten -> sittin (2 substitutions (k turned into s and e turned into i) = fuzziness is 2)

If you search for similar terms (to catch typos for example), use fuzziness.

"The default edit distance is 2, but an edit distance of 1 should be sufficient to catch 80% of all human misspellings. It can be specified as: quikc~1" (source: Elastic).

Example: quikc~ brwn~ foks~ (as the default edit distance is 2, this query will catch all quick, quack, quock, uqikc, etc. as well as brown, folks, etc.)

Example: Datashare~1 (this query will catch Datasahre, Dqtashare, etc.)

What are proximity searches?

When you turn on 'Do phrase matches', you can set, in 'Proximity searches', the maximum number of operations (insertions, deletions, substitutions and transpositions) on terms needed to make one phrase match the other.

“the cat is blue” -> “the small cat is blue” (1 insertion = fuzziness is 1)

“the cat is blue” -> “the small is cat blue” (1 insertion + 2 transpositions = fuzziness is 3)

Example: "fox quick"~5 (this query will catch "quick brown fox", "quick brown car thin fox" or even "quick brown car thin blue tree fox"

• Click 'Add'. Your batch search will appear in the table of batch searches.

Get your results

• Open your batch search by clicking its name:

• You see your results and you can sort them by clicking the column's name. 'Rank' means the order by which each queries would be sorted out if run in Datashare's main search bar. They are thus sorted by relevance score by default.

• You can click on a document's name and it will open it in a new tab:

• You can filter your results by query and read how many documents there are for each query:

You can search for specific queries:

• You can also download your results in a CSV format:

Relaunch your batch search

If you add more and more files in Datashare, you might want to relaunch existing batch search on your new documents too.

Notes:

• In the server collaborative mode, you can only relaunch your own batch searches, not others'.

• The relaunched batch search will apply to your whole corpus, newly indexed documents and previously indexed documents (not only the newly indexed ones).

1. To do so, open the batch search that you'd like to relaunch and click 'Relaunch':

1. Edit the name and the description of your batch search if needed:

1. You can choose to delete the current batch search after relaunching it:

Note: if you're worried about losing your previous results because of an error, we recommend to keep your current batch search (turn off this toggle button) and delete it only after the relaunch is a success.

1. Click 'Submit':

You can see your relaunched batch search running in the batch search's list:

I get a "failure". What does that mean?

Failures in batch searches can be due to several causes.

Click the 'See error' button to open the error window:

The first query containing an error makes the batch search fail and stop.

Check this first failure-generating query in the error window:

In the case above, the slash (/) used between 'Heroin' and 'Opiates' is a reserved character that was not escaped by a backslash so Datashare interpreted this query as a syntax error, failed and didn't go further so the batch search stopped.

We recommend to remove the slash, as well as any reserved characters, and re-run the batch search again.

'elasticsearch: Name does not resolve'

If you have a message which contain 'elasticsearch: Name does not resolve', it means that Datashare can't make Elastic Search, its search engine, work.

In that case, you need to re-open Datashare: ****here are the instructions for Mac, Windows or Linux.

Example of a message regarding a problem with ElasticSearch:

SearchException: query='lovelace' message='org.icij.datashare.batch.SearchException: java.io.IOException: elasticsearch: Name does not resolve'

__

'Data too large'

One of your queries can lead to a 'Data too large' error.

It means that this query had too many results or in their results, some documents that were too big to process for Datashare. This makes the search engine fail.

We recommend to remove the query responsible for the error and re-start your batch search without the query which led to the 'Data too large' error.

****

'SearchException: query='AND ada' '

One or several of your queries contains syntax errors.

It means that you wrote one or more of your queries the wrong way with some characters that are reserved as operators (see below).

You need to correct the error(s) in your CSV and re-launch your new batch search with a CSV that does not contain errors. Click here to learn how to launch a batch search.

Datashare stops at the first syntax error. It reports only the first ​error. You might need to check all your queries as some errors can remain after correcting the first one.

They are more likely to happen when 'do phrase matches' toggle button is turned off:

When 'Do phrase matches' is on, syntax error can still happen though:

Here are the most common errors:

- A query starts with AND (all uppercase)

You cannot start a query with AND all uppercase, neither in Datashare's main search bar nor in your CSV. AND is reserved as a search operator.

- A query starts with OR (all uppercase)

You cannot start a query with OR all uppercase, neither in Datashare's main search bar nor in your CSV. OR is reserved as a search operator.

- A query contains only one double quote or a double quote in a word

You cannot type a query with only one double quote, neither in Datashare's main search bar nor in your CSV. Double quotes are reserved as a search operator.

- A query starts with or contains tilde (~) inside a term

You cannot start a query with tilde (~) or make one contain a tilde, neither in Datashare's main search bar nor in your CSV. Tilde is reserved as a search operator for fuzziness or proximity searches.

- A query starts with or contains a caret (^)

You cannot start a query with caret (^) or make it contain a caret, neither in Datashare's main search bar nor in your CSV. Caret is reserved as a boosting operator.

- A query contains one slash (/)

You cannot start a query with slash (/) or make it contain a slash, neither in Datashare's main search bar nor in your CSV. Slash is a reserved character to open Regex ('regular expressions'). Note that you can use Regex in batch searches.

- A query uses square brackets ([ ])

Delete your batch search

Open your batch search and click the trash icon:

Then click 'Yes':

Star a document

• Hover the document's title in the main column. A button with the labe 'Star document' appears.

• Click it:

The starred document is now marked with a star icon:

Unstar a document

Click again on its star icon. The document won't have a star anymore.

Find your starred documents

• Open the 'Starred' filter on the left:

• Tick 'Starred' (respectively 'Unstarred').

• Only starred (respectively unstarred) documents will be displayed in the main column:

Tag a document

• Open the document by clicking on its title

• Click the second tab 'Tags & Details'

• Type your tag

• Press 'Enter'

Tags can contain any character but cannot contain space.

Your new tag is now displayed on this page.

You can add several tags.

Search tags with tag filter

• Open the second filter untitled 'Tags'

• You see the tags by frequency and the number of tagged documents

• You can search using the search bar

• You can select one or multiple tags

Search tags with main search bar

To find all your documents tagged with specific tag(s):

• Type the tag(s) in the main search bar

• Select 'Tags' in the field dropdown menu

• Click 'Search' or press 'Enter'

The results are all the documents tagged with the tag(s) you typed in the search bar.

To find all your tagged documents, whatever the tags:

• Type nothing in the search bar

• Select 'Tags' in the field selector

• Click 'Search'

The results are all the tagged documents.

Delete a tag

Click the cross at the end of the tag that you want to delete.

Find these shortcuts in Datashare here:

It will open a window which recalls the shortcuts:

Go to the next / previous document

• Windows / Linux

Control + →

Control + ←

• Mac

Command (⌘) + →

Command (⌘) + ←

Find in document...

• Windows / Linux

Control + F

• Mac

Command (⌘) + F

... and go from one occurrence to the next / previous occurrence

• Go to next occurrence

Enter

or

F3

• Go to previous occurrence

Shift + Enter

or

Shift + F3

Navigate a document's tabs

• Windows / Linux

Control (ctrl) + alt + ⇞ (pageup)

Control (ctrl) + alt + ⇟ (pagedown)

• Mac

Command (⌘) + option (⌥) + ↑ (arrow up)

Command (⌘) + option (⌥) + ↓ (arrow down)

Go back to search results

Once you opened a document, go back to search results:

Esc

Yes, you can remove documents from Datashare. But at the moment, it will remove all your documents. You cannot remove only some documents.

Click the pink trash icon on the bottom left of Datashare:

And then click 'Yes':

You can them re-analyze a new corpus.

For advanced users only - if you'd like to do it with the Terminal, here are the instructions:

• If you're using Mac: rm -Rf ~/Library/Datashare/index

• If you're using Windows: rd /s /q "%APPDATA%"\Datashare\index

• If you're using Linux: rm -Rf ~/.local/share/datashare/index

The documents and entities graph

is a graph database technology which lets you represent your data as a graph. Inside Datashare, neo4j lets you connect entities between them through documents in which they appear.

After creating a graph from your Datashare project, you will be able to explore it and visualize these kinds of relationships between you project entities:

In the above graph, we can see 3 email document nodes in orange, 3 email address nodes in red, 1 person node in green and 1 location node in yellow. Reading the relationship types on the arrows, we can deduce the following information from the graph:

• shapp@caiso.com emailed 20participants@caiso.com, the sent email has an id starting with f4db344...

• one person named vincent is mentioned inside this email, as well as the california location

• finally, the email also mentions the dle@caiso.com email address which is also mentioned in 2 other email documents (with id starting with 11df197... and 033b4a2...)

If you are not familiar with graph and neo4j, take a look at the following resources:

Graph nodes

The neo4j graph is composed of :Document nodes representing Datashare documents and :NamedEntity nodes representing entities mentioned in these documents.

The :NamedEntity nodes are additionally annotated with their entity types: :NamedEntity:PERSON, :NamedEntity:ORGANIZATION, :NamedEntity:LOCATION, :NamedEntity:EMAIL...

Graph relationships

In most cases, an entity :APPEARS_IN a document, which means that it was detected in the document content. In the particular case of email documents and EMAIL addresses, it is most of the time possible to identify richer relationships from the email metadata, such as who sent (:SENT relationship) and who received (:RECEIVED relationship) the email.

When an :EMAIL address entity is neither :SENT or :RECEIVED, like it is the case in the above graph for dle@caiso.com, it means that the address was mentioned in the email document body.

When a document is embedded inside another document (as an email attachment for instance), the child document is connected to its parent through the :HAS_PARENT relationship.

Create your Datashare project's graph

The creation of a neo4j graph inside Datashare is supported through a plugin. To use the plugin to create a graph, follow these instructions:

After the graph is created, navigate to the 'Projects' page and select your project. You should be able to visualize a new neo4j widget displaying the number of documents and entities found inside the graph:

Access your project's graph

Exporting and importing the graph into your own DB is also useful when you want to perform write operations on your graph without any consequences on Datashare.

With read access to Datashare's neo4j database

Without read access to Datashare's neo4j database

If you can't have read access to the database, you will need to export it and import it into your own neo4j instance (running on your laptop for instance).

Ask for a DB dump

Export your graph from Datashare

In case you don't have access to the DB and can't be provided with a dump, you can export the graph from inside. Be aware that limits might be applied on the size of the exported graph.

To export the graph, navigate to Datashare's 'Projects' page, select your project, select the 'Cypher shell' export format and click the 'Export graph' button:

In case you want to restrict the size of the exported graph, you can restrict the export to a subset of documents and their entities using the 'File types' and 'Project directory' filters.

DB import

Docker

• identify your neo4j instance container ID:
• copy your the graph dump inside your neo4j container import directory:

Neo4j Desktop import

• open 'Cypher shell':

• copy your the graph dump inside your neo4j instance import directory:

You will now be able to explore the graph imported in your own neo4j instance.

Explore and visualize entity links

Connect to your database

Visualize and explore with Neo4j Bloom

Neo4j Bloom is accessible from inside Neo4j Desktop app.

Find out more information about to use Neo4j Bloom to explore your graph with:

Query the graph with Neo4j Browser

The Neo4j Browser is available for both Enterprise and Community distributions. You can access it:

Visualize and explore with Linkurious Enterprise Explorer

Find out more information about Linkurious:

Visualize with Gephi

Find out more information about:

Export your graph in the GraphML format

In case you want to restrict the size of the exported graph, you can restrict the export to a subset of documents and their entities using the 'File types' and 'Project directory' filters.

You can download a document by going to it on Datashare. Click on the download icon to the right of the screen on on the right of the document's title.

If you can't download a document, it means that Datashare has been badly initialized. Please restart Datashare. If you're an advanced user, you can capture the logs and create an issue on .

You can use Datashare with multiple users accessing a centralized database on a server.

Warning: to put the server mode in place and to maintain it requires some technical knowledge.

Please find the documentation here.

Warning: this requires some technological knowledge.

You can make Datashare follow soft links : add --followSymlinks when Datashare is launched.

If you're on Mac or Windows, you must change the launch script.

If you're on Linux, you can add the option after the Datashare command.

Datashare was created with scalability in mind which gave ICIJ the ability to index terabytes of documents.

To do so, we used a cluster of dozens of EC2 instances on AWS, running on Ubuntu 16.04 and 18.04. We used c4.8xlarge instances (36 CPUs / 60 GB RAM).

The most complex operation is OCR (we use Apache Tesseract) so if your documents don't contain many images, it might be worth deactivating it (--ocr false).

Tarentula is a tool made for advanced users to run bulk actions in Datashare, like:

• Clean Tags by Query

• Download

• Export by Query

• Tagging

• CSV formats

• Tagging by Query

Please find all the use cases in Datashare Tarentula's GitHub documentation.

Run neo4j inside docker

1. enrich the services section of the docker-compose.yml of the install with Docker page, with the following neo4j service:

...
services:
    neo4j:
      image: neo4j:5-community
      environment:
        NEO4J_AUTH: none
        NEO4J_PLUGINS: '["apoc"]'
      ports:
        - 7474:7474
        - 7687:7687
      volumes:
        - neo4j_conf:/var/lib/neo4j/conf
        - neo4j_data:/var/lib/neo4j/data

make sure not to forget the APOC plugin (NEO4J_PLUGINS: '["apoc"]').

2. enrich the volumes section of the docker-compose.yml of the install with Docker page, with the following neo4j volumes:

volumes:
  ...
  neo4j_data:
    driver: local
  neo4j_conf:
    driver: local

3. Start the neo4j service using:

docker compose up -d neo4j

Run Neo4j Desktop

1. install with Neo4j Desktop, follow installation instructions found here

2. create a new local DBMS and save your password for later

3. if the installer notifies you of any ports modification, check the DBMS settings and save the server.bolt.listen_address for later

4. make sure to install the APOC Plugin

Additional options

Additional options to install neo4j are listed here.

You can send an email to datashare@icij.org.

When reporting a bug, please share:

• your OS (Mac, Windows or Linux) and version

• the problem, with screenshots if possible

• the actions that led to the problem

Advanced users can post an issue with their logs on Datashare's GitHub : https://github.com/ICIJ/datashare/issues

If you search "Shakespeare" in the search bar and if you run a query containing "Shakespeare" in a batch search, you can get slightly different documents between the two results.

Why?

For technical reasons, Datashare processes both queries in 2 different ways:

a. Search bar (a simple search processed in the browser):

The search query sent to Elasticsearch is processed in your browser by Datashare's client. It is then sent to Elasticsearch through Datashare server which forwards your query.

b. Batch search (several searches processed by the server):

1. Datashare's server processes each of the batch search's queries

2. Each query is sent to Elasticsearch. The results are saved into a database

3. When the batch search is finished, you get the results from Datashare

4. Datashare sends back the results stored into the database/

Datashare's team attempts to make both results be similar, but slight differences can happen between the two queries.

Mac

1. Go to Applications

2. Click right on 'Datashare' and click 'Move to Bin'

Windows

Follow the steps here: https://support.microsoft.com/en-us/windows/uninstall-or-remove-apps-and-programs-in-windows-10-4b55f974-2cc6-2d2b-d092-5905080eaf98

Linux

Use the following command:

sudo apt remove datashare-dist

When you open your desktop, you will see a folder called 'Datashare Data'. Move or copy and paste the documents you want to add to Datashare to this folder:

Now open Datashare, which you will find in your main menu (see above: 'Open Datashare')

Once Datashare has opened, click on 'Analyze documents' on the top navigation bar in Datashare:​

You're now ready to analyze your documents in Datashare.

1. You can find a folder called 'Datashare' in your home directory.

2. Move the documents you want to add to Datashare into this folder.

3. Open Datashare to extract text and eventually find people, organizations and locations in your documents.

You can now .

In Datashare, for technical reasons, it is not possible to open the 10,000th document.

Example: you search for "Paris", you get 15,634 results. You'd be able to see the first 9,999th results but no more. This also happens if you didn't run any search.

As it is not possible to fix this, here are some tips:

• : use filters to narrow down your results and ensure you have less than 10,000 matching documents

• Change the : use 'creation date' or 'alphabetical order' for instance, instead of the sorting by default which corresponds to a relevance scoring

• Search your query in a : you will get all your results either on the batch search results' page or, by downloading your results, in a spreadsheet. From there, you will be able to open and read all your documents