To report a bug, please contribute in our GitHub detailing your logs with:

• your Operating System (Mac, Windows or Linux)

• the version of your Operating System

• the version of Datashare

• screenshots of your issue

• a description of your issue.

If for confidentiality reasons you don't want to open an issue on Github, please write to datashare@icij.org and our team will do its best to answer you in a timely manner.

In local mode, Datashare provides a self-contained software application that users can install and run on their own local machines. The software allows users to search into their documents within their own local environments, without relying on external servers or cloud infrastructure. This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.

What is Datashare?

Welcome to Datashare - a self-hosted documents search software. It is a free and open-source software developed by the International Consortium of Investigative Journalists (ICIJ). Initially created to combine multiple named-entity recognition pipelines, this tool is now a fully-featured search interface to dig into your documents. With the help of several open source tools (Extract, Apache Tika, Apache Tesseract, CoreNLP, OpenNLP, Elasticsearch, etc), Datashare can be used on one single personal computer as well as on 100 interconnected servers.

Who uses it?

Datashare is developed by the ICIJ, a collective of investigative journalists. Datashare is built at the top of technologies and methods already tested with investigations like the Panama Papers or the Luanda Leaks. Seeing the growing interest for ICIJ's technology, we decided to open source this key component of our investigations so a single journalist as well as big media organizations could use it on their own documents.

Curious to know more about how we use Datashare?

• How ICIJ analysed 715,000 Luanda Leaks records

• Help test and improve our latest journalism tool

• How Datashare project will help journalists breach borders

Where can I see it in action?

We setup a Demo instance of Datashare with a small set of documents from the Luxleaks investigation (2014). When using this instance, you will be assigned a temporary user which can star, tag, recommend and explore documents.

Can I run it on my server?

Datashare was also built to run on a server. This is how we use it for our collaborative projects. Please refer to the server documentation to know how it works.

Can I customize Datashare?

When building Datashare, one of our first decisions was to use Elasticsearch to create an index of documents. It would be fair to describe Datashare as a nice looking web interface for Elasticsearch. We want our search platform to be user-friendly while keeping all the powerful Elasticsearch features available for advanced users. This way we ensure that Datashare is usable by non tech-savvy reporters, but still robust enough to satisfy data analysts and developers who want to query the index directly with our API.

We implemented the possibility to create plugins, to make this process more accessible. Instead of modifying Datashare directly, you could isolate your code with a specific set of features and then configure Datashare to use it. Each Datashare user could pick the plugins they need or want, and have a fully customized installation of our search platform. Please have a look at the documentation.

Translations

This project is currently available in English, French, Spanish and Japanese. You can help us to improve and complete translations on Crowdin.

The CLI stages are primarly intented to be run for an instance of Datashare that uses non-embedded resources (ElasticSearch, database, key/value memory store). This allows you to distribute heaving tasks between servers.

1. SCAN

This is the first step to add documents to Datashare from the command-line. The SCAN stage allows you to queue all the files that need to be indexed (next step). Once this task is done, you can move to the next step. This stage cannot be distributed.

2. INDEX

3. NLP

Once a document is available for search (stored in ElasticSearch), you can use the NLP stage to extract named entities from the text. This process will not only create named entities mentions in ElasticSearch, it will mark every analyzed document with the corresponding NLP pipeline (CORENLP by default). In other words, the process is idempotent and can be paralelized as well on several servers.

Web modes

Those two modes are the only one who create

In local mode and embedded mode, Datashare provides a self-contained software application that users can install and run on their own local machines. The software allows users to search into their documents within their own local environments, without relying on external servers or cloud infrastructure. This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.

In server mode, Datashare operates as a centralized server-based system. Users can access to the platform through a web interface, and the documents are stored and processed on Datashare's servers. This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.

Comparaison between modes

The running modes offer advantages and limitations. This matrix summarizes the differences:

When running Datashare in local mode, users can choose to use embedded services (like ElasticSearch, SQLITE, in-memory key/value store) on the same JVM than Datashare. This variant of the local mode is called "embedded mode" and allows user to run Datashare without having to setup any additional software. The embedded mode is used by default.

CLI mode

In cli mode, Datashare starts without a web server and allow user to perform task over their documents. This mode can be used in conjunction both with local and server modes, while allowing users to distribute heaving task between several servers.

Daemon modes

Those modes are intended to be used for action that requires to "wait" for pendings tasks.

In batch download mode, the daemon wait for a user to request a batch download of documents. When a request is receive, the daemon start a task to download the document matching the user search, a bundle them into a zip file.

In batch search mode, the daemon wait for a user to request a batch search of documents. To create a batch search, users must go through the dedicated form on Datashare where they can to upload a list of search terms (in CSV format). The daemon will then start a task to search all matching document and store every occurences in the database.

How to change modes

Datashare is shipped as a single executable, with all modes available. As previously mentioned, the default mpode is the embedded mode. Yet when starting Datashare in command line, you can explicitely specify the running mode. For instance on Ubuntu/Debian:

The installer will setup:

• MacPorts (if neither Homebrew or MacPorts are installed)

• Apache Tesseract with MacPorts or Homenrew

• Java JRE 17

• Datashare executable

1. Download Datashare

2. Start the installer

Go to your "Downloads" directory in Finder and double-click "datashare-X.Y.Z.pkg":

3. Go through the Datashare Installer

Click 'Continue', 'Install', enter your password and 'Install Software':

The installation begins. You see a progress bar. It stays a long time on "Running package scripts" because it is installing XCode Command Line Tools, MacPort, Tesseract OCR, Java Runtime Environment and finally Datashare.

You can see what it actually does by typing command+L, it will open a window which logs every action made.

In the end, you should see this screen:

When you open your desktop, you will see a folder called 'Datashare Data'. Move or copy and paste the documents you want to add to Datashare to this folder:

Now open Datashare, which you will find in your main menu (see above: 'Open Datashare')

Once Datashare has opened, click on 'Analyze documents' on the top navigation bar in Datashare:​

You're now ready to analyze your documents in Datashare.

Find your Datashare folder on your Mac

Open your Mac's 'Finder' by clicking on the blue smiling icon in your Mac's 'Dock':

On the menu bar at the top of your computer, click 'Go'. Click on 'Home' (the house icon).

You will see a folder called 'Datashare':

If you want to quickly access it in the future, you can drag and drop it in 'Favorites' on the left of this window:

Add documents in your Datashare folder

Copy or place the documents you want to have in Datashare in this Datashare folder.

Launch Datashare

Open your Applications. You should see Datashare. Double click on it:

Datashare opens in your default internet browser. Click 'Tasks':

Click the 3rd tab 'Analyze your documents':

You can now analyze your documents in Datashare.

Open the Windows main menu at the left of the bar at the bottom of your computer screen and click on 'Datashare'. (The numbers after 'Datashare' just indicate which version of Datashare you installed.)

A window called 'Terminal' will have opened, showing the progress of opening Datashare. Do not close this black window as long as you use Datashare.

Keep this Terminal window open as long as you use Datashare.

Once Datashare is installed, go to "Finder", then "Applications", and double-click on "Datashare".

A Terminal window called 'Datashare.command' opens and describes the technical operations going on during the opening.

Keep this Terminal window open as long as you use Datashare.

Standard (Windows 7 Service Pack 2 or newer version)

Uninstall any prior standard version

Download Datashare

The file "datashare-X.Y.Z.exe" is now downloaded. Double click on the name of the file in order to execute it.

Allow Datashare to install

As Datashare is not signed, this popup asks for your permission. Don't click 'Don't run' but click 'More info':

Click 'Run anyway':

It asks if you want to allow the app to make changes to your device. Click 'Yes':

Install Datashare

On the Installer Wizard, as you need to download and install OpenJDK11 if it is not installed on your device, click 'Install':

The following windows with progress bars will be displayed:

Choose a language and click 'OK':

Install Tesseract OCR

To install Tesseract OCR, click the following buttons on the Installer Wizard's windows:

Untick 'Show README' and click 'Finish':

Finally, click "Close" to close the installer of TesseractOCR.

Install Datashare.jar

It now downloads the back end and the front end, Datashare.jar:

When it is finished, click 'Close':

Currently, only a .deb package for Debian/Ubuntu is provided.

If you want to run it with another Linux distribution, you can download the latest version of the Datashare jar here : https://github.com/ICIJ/datashare/releases/latest

And adapt the following launch script : https://github.com/ICIJ/datashare/blob/master/datashare-dist/src/main/deb/bin/datashare to your environment.

1. Download Datashare

Go to datashare.icij.org, scroll down and click 'Download .deb'

Save the Debian package as a file

2. Install the package

$ sudo apt install /dir/to/debian/package/datashare-dist_7.2.0_all.deb

3. Run Datashare with:

$ datashare

You can now start Datashare!

Prerequisites

Datashare platform is designed to function effectively by utilizing a combination of various services. To streamline the development and deployment workflows, Datashare relies on the use of Docker containers. Docker provides a lightweight and efficient way to package and distribute software applications, making it easier to manage dependencies and ensure consistency across different environments.

Read more about how to install Docker on your system.

Starting Datashare with a single container

To start Datashare within a Docker container, you can use this command:

docker run --mount src=$HOME/Datashare,target=/home/datashare/data,type=bind -p 8080:8080 icij/datashare:11.1.9 --mode EMBEDDED

Make sure the Datashare folder exists in your homedir or this command will fail. This is an example about how to use Datashare with Docker, data will not be persisted.

Starting Datashare with multiple containers

Within Datashare, Docker Compose can play a significant role in enabling the setup of separated and persistent services for essential components such as the database (PostgreSQL), the search index (Elasticsearch), and the key-value store (Redis).

By utilizing Docker Compose, you can define and manage multiple containers as part of a unified service. This allows for seamless orchestration and deployment of interconnected services, each serving a specific purpose within the Datashare ecosystem.

Specifically, Docker Compose allows you to configure and launch separate containers for PostgreSQL, Elasticsearch, and Redis. These containers can be set up in a way that ensures their data is persistently stored, meaning that any information or changes made to the database, search index, or key-value store will be retained even if the containers are restarted or redeployed.

This separation of services using Docker Compose provides several advantages. It enhances modularity, scalability, and maintainability within the Datashare platform. It allows for independent management and scaling of each service, facilitating efficient resource utilization and enabling seamless upgrades or replacements of individual components as needed.

To start Datashare with Docker Compose, you can use the following docker-compose.yml file:

version: "3.7"
services:

  datashare:
    image: icij/datashare:18.1.3
    hostname: datashare
    ports:
      - 8080:8080
    environment:
      - DS_DOCKER_MOUNTED_DATA_DIR=/home/datashare/data
    volumes:
      - type: bind
        source: ${HOME}/Datashare
        target: /home/datashare/data
      - type: volume
        source: datashare-models
        target: /home/datashare/dist
    command: >-
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password 
      --mode LOCAL
      --tcpListenPort 8080
    depends_on:
      - postgresql
      - redis
      - elasticsearch

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
    restart: on-failure
    volumes:
      - type: volume
        source: elasticsearch-data
        target: /usr/share/elasticsearch/data
        read_only: false
    environment:
      - "http.host=0.0.0.0"
      - "transport.host=0.0.0.0"
      - "cluster.name=datashare"
      - "discovery.type=single-node"
      - "discovery.zen.minimum_master_nodes=1"
      - "xpack.license.self_generated.type=basic"
      - "http.cors.enabled=true"
      - "http.cors.allow-origin=*"
      - "http.cors.allow-methods=OPTIONS, HEAD, GET, POST, PUT, DELETE"

  redis:
    image: redis:4.0.1-alpine
    restart: on-failure

  postgresql:
    image: postgres:12-alpine
    environment:
      - POSTGRES_USER=datashare
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=datashare
    volumes:
      - type: volume
        source: postgresql-data
        target: /var/lib/postgresql/data

volumes:
  datashare-models:
  elasticsearch-data:
  postgresql-data:

Open a terminal or command prompt and navigate to the directory where you saved the docker-compose.yml file. Then run the following command to start the Datashare service:

docker-compose up -d

The -d flag runs the containers in detached mode, allowing them to run in the background.

Docker Compose will pull the necessary Docker images (if not already present) and start the containers defined in the YAML file. Datashare will take a few seconds to start. You can check the progression of this opperation with:

docker-compose logs -f datashare

Once the containers are up and running, you can access the Datashare service by opening a web browser and entering http://localhost:8080. This assumes that the default port mapping of 8080:8080 is used for the Datashare container in the YAML file.

That's it! You should now have the Datashare service up and running, accessible through your web browser. Remember that the containers will continue to run until you explicitly stop them.

To stop the Datashare service and remove the containers, you can run the following command in the same directory where the docker-compose.yml file is located:

docker-compose down

This will stop and remove the containers, freeing up system resources.

Start Datashare by launching it from the command-line:

datashare

Datashare should now automatically open in your default internet browser. If it doesn’t, type "localhost:8080" in your browser. Datashare must be accessed from your internet browser (Firefox, Chome, etc), even though it works offline without Internet connection (see: Can I use Datashare with no internet connection?).

It's now time to add documents to Datashare.

To be able to perform OCR, Datashare uses an open source technology called Apache Tesseract. When Tesseract extracts text from images, it uses "language packages" especially trained for each specific languages. Unfortunately, those packages can be heavy and to ensure a lightweight installation of Datashare, the installer doesn't use them all by default. In the case Datashare informs you of a missing package, this guide will explain you how to manually install it on your system.

Install packages on Linux

To add ocr languages on linux, simply use the following command:

Where `[lang]` is can be :

• all if you want to install all languages

Install packages on Mac

With MacPorts (default)

First, you must check that MacPort is installed on your computer. Please run in a Terminal:

You should see an output similar to this:

If MacPort is installed on your computer, you should be able to add the missing Tesseract language package with the following command (for German):

Once the installation is done, simply close and restart Datashare to be able to use the newly installed packages.

With Homebrew

If Homebrew was already present on your system when Datashare was installed, Datashare used it to install Tesseract and its language packages. Because Homebrew doesn't package each Tesseract language individually, all languages are already supported by your system. In other words, you have nothing to do!

If you want to check if Homebrew is installed, run the following command in a Terminal:

You should see an output similar to this:

Install languages on Windows

*Additionnal languages can be also added during Tesseract's installation.

The list of installed languages can be checked with Windows command prompt or Powershell with the commandtesseract --list-langs.

Datashare has to be restarted after the language installation.

Add documents

1. To add your documents in Datashare, click 'Tasks' in the left menu:

2. Click 'Analyze your documents':

3. Click 'Add documents' so Datashare can extract the texts from your files:

Options when adding documents

You can:

• Select the specific folder or sub-folder containing the documents you want to add.

• Extract text also from images/PDFs (OCR). Be aware the indexing can be up to 10 times longer.

• Skip already indexed files.

Two extraction tasks are now running: the first is the scanning of your Datashare folder which sees if there are new documents to analyze (ScanTask). The second is the indexing of these files (IndexTask):

It is not possible to 'Find people, organizations and locations' while of these two tasks is still running.

When tasks are done, you can start exploring documents by clicking 'Search' in the left menu but you won't have the named entities (names of people, organizations and locations) yet. To have these, follow the steps below.

Extract names of people, organizations and locations

1. After the text is extracted, you can launch named entities recognition by clicking the button 'Find people, organizations and locations'.

2. In the window below, you are asked to choose between finding Named Entities or finding email addresses (you cannot do both simultaneously, you need to do one after the other, no matter the order):

You can now see running tasks and their progress. After they are done, you can click 'Clear done tasks' to stop displaying tasks that are completed.

3. You can search your indexed documents without having to wait for all tasks to be done. To access your documents, click 'Search':

Extract email addresses

To extract email addresses in your documents:

• Re-click on 'Find people, organizations, locations and email addresses' (in Tasks (left menu) > Analyze your documents)

• Click the second radio button 'Find email addresses':

Prerequisites

Get neo4j up and running

Follow the instructions of the dedicated FAQ page to get neo4j up and running.

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the 'Other platforms and versions' button when downloading to access versions if necessary.

Add entities

If it's not done yet analyze your documents and extract both names of people, organizations and locations as well as email addresses.

If your project contains email documents, make sure to also extract email addresses.

Next step

You can now run Datashare with the neo4j plugin !

Plugins are small programs that you can add to Datashare's front-end to get new features (the front-end is the interface, "the part of the software with which the user interacts directly" - source).

Extensions are small programs that you can add to Datashare's back-end to add new features (the back-end is "the part of the software that is not directly accessed by the user, typically responsible for storing and manipulating data" - source).

Add plugins to Datashare (front-end)

1. Go to "Settings":

1. Click "Plugins":

1. Choose the plugin you want to add and click "Install now":

If you want to install a plugin from an URL, click "Install plugin from URL".

1. Your plugin is installed.

Refresh your page to see your new plugin activated in Datashare.

Add extensions to Datashare (back-end)

1. Go to "Settings":

1. Click "Extensions":

1. Choose the extension you want to add and click "Install now":

If you want to install an extension from an URL, click "Install extension from URL".

1. Your extension is installed.

Restart Datashare to see your new extension activated in Datashare.

Update plugin or extension with latest version

When a newer version of a plugin or extension is available, you can click on the "Update" button to get the latest version.

After that, if it is a plugin, refresh your page to activate the latest version.

If it is an extension, restart Datashare to activate the latest version.

Create your own plugin or extension

People who code can create their own plugins and extensions by following these steps:

• Plugins: https://github.com/ICIJ/datashare/wiki/Client-%E2%80%BA-Plugins

• Extensions: https://github.com/ICIJ/datashare/wiki/Backend-%E2%80%BA-Extensions

1. You can find a folder called 'Datashare' in your home directory.

2. Move the documents you want to add to Datashare into this folder.

3. Open Datashare to extract text and eventually find people, organizations and locations in your documents.

You can now analyze your documents.

In server mode, Datashare operates as a centralized server-based system. Users can access to the platform through a web interface, and the documents are stored and processed on Datashare's servers. This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.

Launch configuration

Datashare is launched with --mode SERVER and you have to provide:

• the external elasticsearch index address elasticsearchAddress

• a Redis store address redisAddress

• a Redis data bus address messageBusAddress

• a database JDBC URL dataSourceUrl

• the host of Datashare (used to generate batch search results URLs) rootHost

• an authentication mechanism and its parameters

Example:

docker run -ti ICIJ/datashare:version --mode SERVER \
    --redisAddress redis://my.redis-server.org:6379 \
    --elasticsearchAddress https://my.elastic-server.org:9200 \
    --messageBusAddress my.redis-server.org \
    --dataSourceUrl jdbc:postgresql://db-server/ds-database?user=ds-user&password=ds-password \
    --rootHost https://my.datashare-server.org
    # ... +auth parameters (see authentication providers section)

Create the graph

1. Open the 'Projects' page and select your project:

1. Create the graph by clicking on the 'Create graph' button inside the neo4j widget:

You will see a new import task running:

When the graph creation is complete, 'Graph statistics' will reflect the number of documents and entities nodes found in the graph:

Update the graph

When new documents or entities are added or modified inside Datashare, you will need to update the neo4j graph to reflect these changes.

To update the graph click on the 'Update graph' button inside the neo4j widget:

To detect whether a graph update is needed you can compare the number of documents found inside Datashare to the number found in the 'Graph statistics' and run an update in case of mismatch:

The update will always add missing nodes and relationships, update existing ones if they were modified, but will never delete graph nodes or relationships.

Next steps

• explore your graph using your favorite visualization tool

Install the neo4j plugin

Configure the neo4j plugin

1. Go to "Settings":

2. Make sure the following settings are properly set:

• Neo4j Host should be localhost or the address where your neo4j instance is running

• Neo4j Port should be the port where your neo4j instance is running (7687 by default)

• Neo4j User should be set to your neo4j user name (neo4j by default)

• Neo4j Password should only be set if your neo4j user is using password authentication

3. When running Neo4j Community Edition, set the Neo4j Single Project value. In community edition, the neo4j DBMS is restricted to a single database. Since Datashare supports multiple projects, you must set the Neo4j Single Project with the name of the project which will use neo4j plugin. Other projects won't be able to use the neo4j plugin.

4. Restart Datashare to apply the changes

5. You should be able to see the neo4j widget in your project page, after a little while its status should be RUNNING:

Next step

This is likelly to be changed in the near future, but in the meantime, you can still add documents to Datashare using the command-line interface.

Here is a simple command to scan a directory and index its files:

What's happening here:

• The SCAN stage feeds a queue in memory with file to add

• The INDEX stage pulls files from the queue to add them to ElasticSearch

• We tell Datashare to use the elasticsearch service

• Files to add are located in /home/datashare/Datashare/ which is a directory mounted from the host machine

Alternativly, you can do this in two separated phases, as long as you tell Datashare to store the queue in a shared resource. Here, we use the redis:

Once the opperation is done, we can easily check the content of queue created by Datashare in redis. In this example we only display the 20 first files in the datashare:queue:

Once the indexing is done, Datashare will exit gracefully and your document will already be visible on Datashare.

Sometimes you will face the case where you have an existing index, and you want to index additional documents inside your working directory without processing every document again. It can be done in two steps :

• Scan the existing ElasticSearch index and gather document paths to store it inside a report queue

• Scan and index (with OCR) the documents in the directory, thanks to the previous report queue, it will skip the paths inside of it

Prerequisites

Datashare platform is designed to function effectively by utilizing a combination of various services. To streamline the development and deployment workflows, Datashare relies on the use of Docker containers. Docker provides a lightweight and efficient way to package and distribute software applications, making it easier to manage dependencies and ensure consistency across different environments.

Read more about how to install Docker on your system.

Starting Datashare with multiple containers

Within Datashare, Docker Compose can play a significant role in enabling the setup of separated and persistent services for essential components. By utilizing Docker Compose, you can define and manage multiple containers as part of a unified service. This allows for seamless orchestration and deployment of interconnected services, each serving a specific purpose within the Datashare ecosystem.

Specifically, Docker Compose allows you to configure and launch separate containers for PostgreSQL, Elasticsearch, and Redis. These containers can be set up in a way that ensures their data is persistently stored, meaning that any information or changes made to the database, search index, or key-value store will be retained even if the containers are restarted or redeployed.

This separation of services using Docker Compose provides several advantages. It enhances modularity, scalability, and maintainability within the Datashare platform. It allows for independent management and scaling of each service, facilitating efficient resource utilization and enabling seamless upgrades or replacements of individual components as needed.

To start Datashare in server mode with Docker Compose, you can use the following docker-compose.yml file:

version: "3.7"
services:

  # This is the main Datashare service that serves the web interface. 
  # Here it is configured with a "dummy" authentication backend which 
  # creates epehemeral user sessions.
  datashare_web:
    image: icij/datashare:18.1.3
    hostname: datashare
    ports:
      - 8080:8080
    environment:
      - DS_DOCKER_MOUNTED_DATA_DIR=${HOME}/Datashare
    volumes:
      - type: bind
        source: ${HOME}/Datashare
        target: /home/datashare/Datashare
    depends_on:
      postgresql:
        condition: service_healthy
      redis:
        condition: service_healthy
      elasticsearch:
        condition: service_healthy
    command: >-
      --mode SERVER
      --dataDir /home/datashare/Datashare
      --pluginsDir /home/datashare/plugins
      --extensionsDir /home/datashare/extensions
      --authFilter org.icij.datashare.session.YesCookieAuthFilter
      --busType REDIS
      --batchQueueType REDIS
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password 
      --defaultProject secret-project
      --elasticsearchAddress http://elasticsearch:9200      
      --messageBusAddress redis://redis:6379
      --queueType REDIS
      --redisAddress redis://redis:6379  
      --rootHost http://localhost:8080
      --sessionStoreType REDIS
      --sessionTtlSeconds 43200
      --tcpListenPort 8080

  # We use a service to create the "secret-project". In theory you only need 
  # to run it the first time you start Datashare. 
  datashare_create_project:
    image: icij/datashare:18.1.3
    restart: no
    depends_on:
      elasticsearch:
        condition: service_healthy
    command: >-
      --defaultProject secret-project 
      --mode CLI 
      --stage INDEX 
      --elasticsearchAddress http://elasticsearch:9200

  # This service starts a deamon that wait for background tasks
  # so it can run them (and save them in the database).
  datashare_task:
    image: icij/datashare:18.1.3
    depends_on:
      - datashare_web
    command: >-
      --mode TASK_RUNNER
      --batchQueueType REDIS
      --batchThrottleMilliseconds 500
      --busType REDIS
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password
      --defaultProject secret-project 
      --elasticsearchAddress http://elasticsearch:9200  
      --queueType REDIS
      --redisAddress redis://redis:6379
      --scrollSize 100  
      
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
    restart: on-failure
    volumes:
      - type: volume
        source: elasticsearch-data
        target: /usr/share/elasticsearch/data
        read_only: false
    environment:
      - "http.host=0.0.0.0"
      - "transport.host=0.0.0.0"
      - "cluster.name=datashare"
      - "discovery.type=single-node"
      - "discovery.zen.minimum_master_nodes=1"
      - "xpack.license.self_generated.type=basic"
      - "http.cors.enabled=true"
      - "http.cors.allow-origin=*"
      - "http.cors.allow-methods=OPTIONS, HEAD, GET, POST, PUT, DELETE"
    healthcheck:
      test: ["CMD-SHELL", "curl --silent --fail elasticsearch:9200/_cluster/health || exit 1"]

  postgresql:
    image: postgres:12-alpine
    environment:
      - POSTGRES_USER=datashare
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=datashare
      # This is needed by the heathcheck command
      # @see https://stackoverflow.com/a/60194261
      - PGUSER=datashare
    volumes:
      - type: volume
        source: postgresql-data
        target: /var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready", "-U", "datashare", "-d", "datashare"]
  
  redis:
    image: redis:4.0.1-alpine
    restart: on-failure
    volumes:
      - type: volume
        source: redis-data
        target: /data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli", "--raw", "incr", "ping"]

volumes:
  datashare-batchdownload-dir:
  elasticsearch-data:
  postgresql-data:
  redis-data:

Open a terminal or command prompt and navigate to the directory where you saved the docker-compose.yml file. Then run the following command to start the Datashare service:

docker-compose up -d

The -d flag runs the containers in detached mode, allowing them to run in the background.

Docker Compose will pull the necessary Docker images (if not already present) and start the containers defined in the YAML file. Datashare will take a few seconds to start. You can check the progression of this opperation with:

docker-compose logs -f datashare_web

Once the containers are up and running, you can access the Datashare service by opening a web browser and entering http://localhost:8080. This assumes that the default port mapping of 8080:8080 is used for the Datashare container in the YAML file.

To stop the Datashare service and remove the containers, you can run the following command in the same directory where the docker-compose.yml file is located:

docker-compose down

This will stop and remove the containers, freeing up system resources.

Add documents to Datashare

If you reach that point, Datashare is up and running but you will discover very quickly that no documents is available in the search results. Next step: Add documents from the CLI.

Extract named entities

Datashare as the ability to detect email addresses, name of people, organizations and locations. You must perform the named entities extraction in the same fashion than the previous commands. Final step: Add named entities from the CLI.

Install the neo4j plugin

Install the neo4j plugin using the Datashare CLI so that users can access it from the frontend:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --pluginInstall datashare-plugin-neo4j-graph-widget 

Installing the plugin install the datashare-plugin-neo4j-graph-widget plugin inside /home/datashare/plugings and will also install the datashare-extension-neo4j backend extension inside /home/datashare/extensions. These locations can be changed by updating the docker-compose.yml.

Configure the neo4j extension

Update the docker-compose.yml to reflect your neo4j docker service settings.

...
services:
    datashare_web:
      ...
      environment:
        - DS_DOCKER_NEO4J_HOST=neo4j
        - DS_DOCKER_NEO4J_PORT=7687
        - DS_DOCKER_NEO4J_SINGLE_PROJECT=secret-project  # This is for community edition only

If your choose a different neo4j user or set a password for your neo4j user make sure to also set DS_DOCKER_NEO4J_USER and DS_DOCKER_NEO4J_PASSWORD.

When running Neo4j Community Edition, set the DS_DOCKER_NEO4J_SINGLE_PROJECT value. In community edition, the neo4j DBMS is restricted to a single database. Since Datashare supports multiple projects, you must set the DS_DOCKER_NEO4J_SINGLE_PROJECT with the name of the project which will use neo4j plugin. Other projects won't be able to use the neo4j plugin.

Restart Datasahre

After installing the plugin a restart might be needed for the plugin to display:

docker compose restart datashare_web

Next step

You can now create the graph !

Prerequisites

Get neo4j up and running

Follow the instructions of the dedicated FAQ page to get neo4j up and running.

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the 'Other platforms and versions' button when downloading to access versions if necessary.

Add entities

If it's not done yet add entities to your project using the Datashare CLI.

If your project contains email documents, make sure to run the EMAIL pipeline together with regular NLP pipeline. To do so add set the follow nlpp flag to --nlpp CORENLP,EMAIL.

Next step

You can now run Datashare with the neo4j plugin !

Authentication with Datashare in server mode is the most impacting choice that has to be made. It can be one of the followings:

• basic authentication with credentials stored in database (PostgreSQL)

• basic authentication with credentials stored in Redis

• OAuth2 with credentials provided by an identity provider (KeyCloak for example)

• dummy basic auth to accept any user (⚠️ if the service is exposed to internet, it will leak your documents)

This document assumes you have installed Datashare in server mode within Docker and already added documents to Datasharte.

In server mode, it's important to understand that Datashare does not provide an interface to add documents. As there is no build-in roles and permission in Datashare's data model, we have no way to differentiate user to offer admin additional tools.

This is likely to be changed in the near future, but in the meantime, you can extract named entities using the command-line interface.

Datashare as the ability to detect email addresses, name of people, organizations and locations. This process use a Natural Language Processing pipeline called CORENLP. Once your documents have been indexed in Datashare, you can perform the named entities extraction in the same fashion as the previous CLI's stages:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage NLP \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --nlpParallelism 2 \
  --nlpp CORENLP

What's happening here:

• Datashare starts in "CLI" mode

• We ask to process the NLP stage

• We tell Datashare to use the elasticsearch service

• Datashare will pull documents from ElasticSearch directly

• Up to 2 documents will be analyzed in parallel

• Datashare will use the CORENLP pipeline

Datashare will use the output queue from the previous INDEX stage (by default extract:queue:nlp in Redis) that contains all the document ids to be analyzed.

The first time you run this command you will have to wait a little bit because Datashare need to download CORENLP's models which can be big.

You can also use chain the 3 stages altogether:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage SCAN,INDEX,NLP \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --nlpParallelism 2 \
  --nlpp CORENLP \
  --dataDir /home/datashare/Datashare/

As for the previous stages you may want to restore the output queue from the INDEX stage. You can do:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage ENQUEUEIDX,NLP \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --nlpParallelism 2 \
  --nlpp CORENLP

The added ENQUEUEIDX stage will read Elasticsearch index, find all documents that have not already been analyzed by the CORENLP NER pipeline, and put the ids of those documents into the extract:queue:nlp queue.

This is the default authentication mode: if not provided in CLI it will be selected. With OAuth2 you will need a third-party authorization service. The diagram below describes the workflow:

Example

docker run -ti ICIJ/datashare:version --mode SERVER \
    --oauthClientId 30045255030c6740ce4c95c \
    --oauthClientSecret 10af3d46399a8143179271e6b726aaf63f20604092106 \
    --oauthAuthorizeUrl https://my.oauth-server.org/oauth/authorize \
    --oauthTokenUrl https://my.oauth-server.org/oauth/token \
    --oauthApiUrl https://my.oauth-server.org/api/v1/me.json \
    --oauthCallbackPath /auth/callback

Integration with KeyCloak

We made a small demo repository to show how it could be setup.

Basic authentication is a simple protocol that uses the HTTP headers and the browser to authenticate users. User credentials are sent to the server in the header Authorization with user:password base64 encoded:

Authorization: Basic dXNlcjpwYXNzd29yZA==

It is secure as long as the communication to the server is encrypted (with SSL for example).

On the server side, you have to provide a database user inventory. You can launch datashare first with the full database URL, then datashare will automatically migrate your database schema. Datashare supports SQLite and PostgreSQL as back-end databases. SQLite is not recommended for a multi-user server because it cannot be multithreaded, so it will introduce contention on users' DB SQL requests.

Then you have to provision users. The passwords are sha256 hex encoded (for example with bash):

$ echo -n bar | sha256sum
fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9  -

Then you can insert the user like this in your database:

$ psql datashare
datashare=> insert into user_inventory (id, email, name, provider, details) values ('fbar', 'foo@bar.com', 'Foo Bar', 'my_company', '{"password": "fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9", "groups_by_applications":{"datashare":["local-datashare"]}}');

If you use other indices, you'll have to include them in the group_by_applications, but local-datashare should remain. For exammple if you use myindex:

$ psql datashare
datashare=> insert into user_inventory (id, email, name, provider, details) values ('fbar', 'foo@bar.com', 'Foo Bar', 'my_company', '{"password": "fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9", "groups_by_applications":{"datashare":["myindex", "local-datashare"]}}');

Or you can use PostgreSQL import CSV COPY statement if you want to create them all at once.

Then when accessing Datashare, you should see this popup:

Example

Here is an example of launching Datashare with Docker and the basic auth provider filter backed in database:

docker run -ti ICIJ/datashare --mode SERVER \
    --batchQueueType REDIS \
    --dataSourceUrl 'jdbc:postgresql://postgres/datashare?user=<username>&password=<password>' \
    --sessionStoreType REDIS \
    --authFilter org.icij.datashare.session.BasicAuthAdaptorFilter \
    --authUsersProvider org.icij.datashare.session.UsersInDb

You can have a dummy authentication that always accepts basic auth. So you should see this popup:

But then you can enter whatever user or password, it will enter Datashare.

Example

docker run -ti ICIJ/datashare -m SERVER \
  --dataDir /home/dev/data \
    --batchQueueType REDIS \
    --dataSourceUrl 'jdbc:postgresql://postgres/datashare?user=dstest&password=test'\
    --sessionStoreType REDIS \
    --authFilter org.icij.datashare.session.YesBasicAuthFilter

Basic authentication is a simple protocol that uses the HTTP headers and the browser to authenticate users. User credentials are sent to the server in the header Authorization with user:password base64 encoded:

Authorization: Basic dXNlcjpwYXNzd29yZA==

It is secure as long as the communication to the server is encrypted (with SSL for example).

On the server side, you have to provide a user store for Datashare. For now we are using a Redis data store.

So you have to provision users. The passwords are sha256 hex encoded. For example using bash:

$ echo -n bar | sha256sum
fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9  -

Then insert the user like this in Redis:

$ redis-cli -h my.redis-server.org
redis-server.org:6379> set foo '{"uid":"foo", "password":"fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9", "groups_by_applications":{"datashare":["local-datashare"]}}'

If you use other indices, you'll have to include them in the group_by_applications, but local-datashare should remain. For exammple if you use myindex:

$ redis-cli -h my.redis-server.org
redis-server.org:6379> set foo '{"uid":"foo", "password":"fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9", "groups_by_applications":{"datashare":["myindex","local-datashare"]}}'

Then you should see this popup:

Example

Here is an example of launching Datashare with Docker and the basic auth provider filter backed in Redis:

docker run -ti ICIJ/datashare --mode SERVER \
    --batchQueueType REDIS \
    --dataSourceUrl 'jdbc:postgresql://postgres/datashare?user=<username>&password=<password>' \
    --sessionStoreType REDIS \
    --authFilter org.icij.datashare.session.BasicAuthAdaptorFilter \
    --authUsersProvider org.icij.datashare.session.UsersInRedis

Search in documents

1. To see all your documents (you need to have added documents to Datashare and have analyzed them before), click 'Search in documents':

If not collapsed yet, to collapse the left menu in order to gain room, click the 'hamburger menu':

2. Search for specific documents. Type terms in the search bar, press Enter or click 'Search':

IMPORTANT:

• To make your searches more precise, you can search with operators (AND, OR, ....): read more here.

• If you get a message "Your search query is wrong", it is probably because you are misusing one or some reserved characters (like ^ " ? ( [ * OR AND etc). Please refer to this page.

3. You can search in specific fields like tags, title, author, recipient, content, path or thread ID. Click 'All fields' and select your choice in the dropdown menu:

Choose between views (list, grid, table)

Select the view on the top right.

• List:

• Grid:

• Table:

Search within a document

Once a document is opened, you can search for terms in this document:

• Press Command (⌘) + F (on Mac) or Control + F (on Windows and Linux) or click on the search bar above your Extracted Text

• Type what you search for

• Press ENTER to go from one occurrence to the next one

• Presse SHIFT + ENTER to go from one occurrence to the previous one

(To know all the shortcuts in Datashare, please read 'Use keyboard shortcuts'.)

This also counts the number of occurrences of your searched terms in this document:

If you run email extraction and searched for one or several email addresses, if the email adresses are in the email's metadata (recipient, sender or other field), there will be a 'in metadata' label attached to the email addresses:

Double quotes for exact phrase

To have all documents mentioning an exact phrase, you can use double quotes. Use straight double quotes ("example"), not curly double quotes (“example”).

Example: "Alicia Martinez’s bank account in Portugal"

OR (or space)

To have all documents mentioning all or one of the queried terms, you can use a simple space between your queries or 'OR'. You need to write 'OR' with all letters uppercase.

Example: Alicia Martinez

Same search: Alicia OR Martinez

AND (or +)

To have all documents mentioning all the queried terms, you can use 'AND' between your queried words. You need to write 'AND' with all letters uppercase.

Example: Alicia AND Martinez

Same search: +Alicia +Martinez

NOT (or ! or -)

To have all documents NOT mentioning some queried terms, you can use 'NOT' before each word you don't want. You need to write 'NOT' with all letters uppercase.

Example: NOT Martinez

Same search: !Martinez

Same search: -Martinez

Please note that you can combine operators

Parentheses should be used whenever multiple operators are used together and you want to give priority to some.

Example: ((Alicia AND Martinez) OR (Delaware AND Pekin) OR Grey) AND NOT "parking lot"

Wildcards

If you search faithf?l, the search engine will look for all words with all possible single character between the second f and the l in this word. It also works with * to replace multiple characters.

Example: Alicia Martin?z

Example: Alicia Mar*z

Fuzziness

You can set fuzziness to 1 or 2. It corresponds to the maximum number of operations (insertions, deletions, substitutions and transpositions) on characters needed to make one term match the other.

kitten -> sitten (1 substitution (k turned into s) = fuzziness is 1)

kitten -> sittin (2 substitutions (k turned into s and e turned into i) = fuzziness is 2)

Example: quikc~ brwn~ foks~ (as the default edit distance is 2, this query will catch all quick, quack, quock, uqikc, etc. as well as brown, folks, etc.)

Example: Datashare~1 (this query will catch Datasahre, Dqtashare, etc.)

Proximity searches

When you type an exact phrase (in double quotes) and use fuzziness, then the meaning of the fuzziness changes. Now, the fuzziness means the maximum number of operations (insertions, deletions, substitutions and transpositions) on terms needed to make one phrase match the other.

Examples:

"the cat is blue" -> "the small cat is blue" (1 insertion = fuzziness is 1)

"the cat is blue" -> "the small is cat blue" (1 insertion + 2 transpositions = fuzziness is 3)

Example: "fox quick"~5 (this query will catch "quick brown fox", "quick brown car thin fox" or even "quick brown car thin blue tree fox"

Boosting operators

Use the boost operator ^ to make one term more relevant than another. For instance, if we want to find all documents about foxes, but we are especially interested in quick foxes:

Example: quick^2 fox

The default boost value is 1, but can be any positive floating point number. Boosts between 0 and 1 reduce relevance. Boosts can also be applied to phrases or to groups:

Example: "john smith"^2 (foo bar)^4

Regular expressions (Regex)

1. You can use Regex in Datashare. Regular expressions (Regex) in Datashare need to be written between 2 slashes.

Example: /.*..*@.*..*/

The example above will search for any expression which is structured like an email address with a dot between two expressions before the @ and a dot between two expressions after the @ like in 'first.lastname@email.com' for instance.

2. Regex can be combined with standard queries in Datashare :

Example: ("Ada Lovelace" OR "Ado Lavelace") AND paris AND /.*..*@.*..*/

3. You need to escape the following characters by typing a backslash just before them (without space):‌ # @ & < > ~

Example: /.*..*@.*..*/ (the @ was escaped by a backslash \ just before it)

We encourage you to use the AND operator to work around this limitation and make sure you can make your search.

If you're looking for French International Bank Account Number (IBAN) that can or cannot contain space and contain FR followed by numbers and/or letters (it could be FR7630001007941234567890185 ou FR76 3000 4000 0312 3456 7890 H43 for example), you can then search for:

Example: /FR[0-9]{14}[0-9a-zA-Z]{11}/ OR (/FR[0-9]{2}.*/ AND /[0-9]{4}.*/ AND /[0-9a-zA-Z]{11}.*/)

1. Here are a few examples of useful Regex:

• You can search for /Dimitr[iyu]/ instead of searching for Dimitri OR Dimitry OR Dimitru. It will find all the Dimitri, Dimitry or Dimitru.

• You can search for /Dimitr[^yu]/ if you want to search all the words which begin with Dimitr and do not end with either y nor u.

• You can search for /Dimitri<1-5>/ if you want to search Dimitri1, Dimitri2, Dimitri3, Dimitri4 or Dimitri5.

1. Other common Regex examples:

• phone numbers with "-" and/or country code like +919367788755, 8989829304, +16308520397 or 786-307-3615 for instance: /[\+]?[(]?[0-9]{3}[)]?[-\s.]?[0-9]{3}[-\s.]?[0-9]{4,6}/

• credit cards: /(?:4[0-9]{12}(?:[0-9]{3})?|[25][1-7][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35[0-9]{3})[0-9]{11})/

(Advanced) Searches using metadata fields

To find the list of existing metadata fields, go to a document's 'Tags and details' tab, click 'Show more details'.

When you hover the lines, you see a magnifying glass on each line. Click on it and Datashare will look for this field. Here is the one for content language:

Here is the one for 'indexing date' (also called extraction date here) for instance:

So for example, if you are looking for documents that:

• contains term1, term2 and term3

• and were created after 2010

you can use the 'Date' filter or type in the search bar:

term1 AND term2 AND term3 AND metadata.tika_metadata_creation_date:>=2010-01-01

Explanations:

• 'metadata.tika_metadata_creation_date:' means that we filter with creation date

• '>="'means 'since January 1st included'

• '2010-01-01' stands for January 2010 and the search will include January 2010

For other searches:

• '<' will mean 'strictly after (with January 1st excluded)'

• nothing will mean 'at this exact date'

You can search for numbers in a range. Ranges can be specified for date, numeric or string fields amont the ones you can find by clicking the magnifying glass when you hover the fields in a document's tab 'Tags and Details'.

Separate Processing Stages

Execute the SCAN and INDEX stages independently to optimize resource allocation and efficiency.

Examples:

Distribute the INDEX Stage

Leverage Parallelism

Datashare offer --parallelism and --parserParallelism options to enhance processing speed.

Example (for g4dn.8xlarge with 32 CPUs):

Optimize ElasticSearch

ElasticSearch can significantly consume CPU and memory, potentially becoming a bottleneck. For production instance of Datashare, we recommend deploying ElasticSearch on a remote server to improve performances.

Adjust JAVA_OPTS

You can fine-tune the JAVA_OPTS environment variable based on your system's configuration to optimize Java Virtual Machine memory usage. Example (for g4dn.8xlarge8with 120 GB Memory):

Specify Document Language

If the document language is known, explicitly setting it can save processing time.

• Use --language for general language setting (e.g., FRENCH, ENGLISH).

• Use --ocrLanguage for OCR tasks to specify the Tesseract model (e.g., fra, eng).

Example:

Manage OCR Tasks Wisely

OCR tasks are resource-intensive. If not needed, disabling OCR can significantly improve processing speed. You can disable OCR with --ocr false.

Example:

Efficient Handling of Large Files

Large PST files or archives can hinder processing efficiency. We recommend extract these files before processing with Datashare. If they are too many of them, keep in mind Datashare will be able to extract them anyway.

Run the neo4j extension CLI

The neo4j related features are added to the DatashareCLI through the extension mechanism. In order to run the extended CLI, the Java CLASSPATH must be extended with the path of the datashare-extension-neo4j jar. By default, this jar is located in /home/datashare/extensions, so the CLI will be run as following:

Create the graph

In order to create the graph, run the --fullImport command for your project:

the CLI will display the import task progress and log import related information.

Update the graph

When new documents or entities are added or modified inside Datashare, you will need to update the neo4j graph to reflect these changes.

To update the graph, you can just re-run the full export:

The update will always add missing nodes and relationships, update existing ones if they were modified, but will never delete graph nodes or relationships.

To detect whether a graph update is needed, open the 'Projects' page and select your project:

compare the number of documents and entities found inside Datashare:

to the numbers found in the 'Graph statistics' and run an update in case of mismatch:

Next steps

Apply filters

On the left column, you can apply filters by ticking them, like 'Portable Document Format (PDF)' in File Types and 'English' in Languages in the example below:

A reminder of the currently applied filters, as well as your queried terms, are displayed at the top of the results' column. You can easily unselect these filters from there by clicking them or clear all of them:

The currently available filters are:

• Projects: if you have more than one project, you can select several of them and run searches in multiple projects at once.

• Starred: If you have starred documents, you can easily find them again.

• Tags: If you wrote some tags, you will be able to select and search for them.

• Recommended by: available only on server (collaborative) mode, this functionality helps you find the document recommended by you and/or others.

• File type: This is the 'Content type' of the file (Word, PDF, JEPG image, etc.) as you can read it in a document's 'Tags & Details'.

• Creation dates: the calendar allows you to select a single creation date or a date range. This is when the document was created as it is noticed in their properties. You can find this in a document's 'Tags & Details'.

• Languages: Datashare detects the main language of each document.

• People / Organizations / Locations: you can selected these named entities and search them.

• Path: This is the location of your documents as it is indicated in your original files (ex: desktop/importantdocuments/mypictures). You can find this in a document's 'Tags & Details'.

• Indexing date: This date corresponds to when you indexed the documents in Datashare.

• Extraction level: This regards embedded documents. The file on disk is level zero. If a document (pictures, etc) is attached or contained in a file on disk, extraction level is “1st”. If a document is attached or contained in a document itself contained in a file on disk, extraction level is “2nd”, etc.

Filters can be combined together and combined with searches in order to refine results.

Filter by named entities

If you have asked Datashare to 'Find people, organizations and locations', you can see names of individuals, organizations and locations in the filters. These are the named entities automatically detected by Datashare.

Search for named entities in the filter's search bar:

Select all of them, one or several of them to filter the documents that mention them:

Use the 'Exclude' button

If you want to select all items except one or several of them, you can use the 'Exclude button'.

It allows you to search for all documents which do not correspond to the filter(s) you selected, that is to say to the currently strikethrough filters.

Contextualize filters

In several filters, you can tick 'Contextualize' : this will update the number of documents indicated in the filters in order to reflect the results. The filter will only count what you selected.

In the example below, the 'Contextualize' checkboxes are not ticked:

After the Contextualize button in Tags filter is ticked:

After the Languages button in Tags filter is ticked:

Clear all filters

To reset all filters at the same time, click 'Clear all':

If you want to search a list of queries in Datashare, instead of doing each of them one by one, you can upload the list directly in Datashare. To do so, you will:

• Create a list of terms that you want to search in the first column of a spreadsheet

• Export the spreadsheet as a CSV (a special format available in any spreadsheet software)

• Upload this CSV in the "new Batch Search" form in Datashare

• Get the results for each query in Datashare - or in a CSV.

Prepare your batch search

Write your queries in a spreadsheet

• Write your queries, one per line and per cell, in the first column of a spreadsheet (Excel, Google Sheets, Numbers, Framacalc, etc.). In the example below, there are 4 queries:

• Do not put line break(s) in any of your cells.

To delete line break(s) in your spreadsheet, you can use the "Find and replace all" functionality. Find all "\n" and replace them all by nothing or a space.

• Write 2 characters minimum in the cells. If one cell contains one character but at least one other cell contains more than one, the cell containing one character will be ignored. If all cells contain only one character, the batch search will lead to 'failure'.

• If you have blank cells in your spreadsheet...

...the CSV (which stand for 'Comma-separated values') will keep these blank cells. It will separate them with semicolons (the 'commas'). You will thus have semicolons in your batch search results (see screenshot below). To avoid that, you need to remove blank cells in your spreadsheet before exporting it as a CSV.

• If there is a comma in one of your cells (like in "1,8 million" in our example above), the CSV will formally put the content of the cell in double quotes in your results and search for the exact phrase in double quotes.

Want to search only on some documents?

For instance, if you want to search only in some documents with certain tag(s), you can write your queries like this: "Paris AND (tags:London OR tags:Madrid NOT tags:Cotonou)".

Use operators in your CSV

Please also note that searches are not case sensitive: if you search 'HeLlo', it will look for all occurrences of 'Hello', 'hello', 'hEllo', 'heLLo', etc. in the documents.

Export your CSV encoded in UTF-8

Export your spreadsheet in a CSV format like this:

• LibreOffice Calc: it uses UTF-8 by default. If not, go to LibreOffice menu > Preferences > Load/Save > HTML Compatibility and make sur the character set is 'Unicode (UTF-8)':

• Google Sheets: it uses UTF-8 by default. Just click "Export to" and "CSV".

• Other spreadsheet softwares: please refer to each software's user guide.

Launch your batch search

• Open Datashare, click 'Batch searches' in the left menu and click 'New batch search' on the top right:

• Type a name for your batch search:

• Upload your CSV:

• Add a description (optional):

• Set the advanced filters ('Do phrase matches', 'Fuzziness' or 'Proximity searches', 'File types' and 'Path') according to your preferences:

What is 'Do phrase matches'?

'Do phrase matches' is the equivalent of double quotes: it looks for documents containing an exact sentence or phrase. If you turn it on, all queries will be search for their exact mention in documents as if Datashare added double quotes around each query.

What is fuzziness?

kitten -> sitten (1 substitution (k turned into s) = fuzziness is 1)

kitten -> sittin (2 substitutions (k turned into s and e turned into i) = fuzziness is 2)

If you search for similar terms (to catch typos for example), use fuzziness.

Example: quikc~ brwn~ foks~ (as the default edit distance is 2, this query will catch all quick, quack, quock, uqikc, etc. as well as brown, folks, etc.)

Example: Datashare~1 (this query will catch Datasahre, Dqtashare, etc.)

What are proximity searches?

When you turn on 'Do phrase matches', you can set, in 'Proximity searches', the maximum number of operations (insertions, deletions, substitutions and transpositions) on terms needed to make one phrase match the other.

“the cat is blue” -> “the small cat is blue” (1 insertion = fuzziness is 1)

“the cat is blue” -> “the small is cat blue” (1 insertion + 2 transpositions = fuzziness is 3)

Example: "fox quick"~5 (this query will catch "quick brown fox", "quick brown car thin fox" or even "quick brown car thin blue tree fox"

• Click 'Add'. Your batch search will appear in the table of batch searches.

Get your results

• Open your batch search by clicking its name:

• You see your results and you can sort them by clicking the column's name. 'Rank' means the order by which each queries would be sorted out if run in Datashare's main search bar. They are thus sorted by relevance score by default.

• You can click on a document's name and it will open it in a new tab:

• You can filter your results by query and read how many documents there are for each query:

You can search for specific queries:

• You can also download your results in a CSV format:

Relaunch your batch search

If you add more and more files in Datashare, you might want to relaunch existing batch search on your new documents too.

Notes:

• In the server collaborative mode, you can only relaunch your own batch searches, not others'.

• The relaunched batch search will apply to your whole corpus, newly indexed documents and previously indexed documents (not only the newly indexed ones).

1. To do so, open the batch search that you'd like to relaunch and click 'Relaunch':

1. Edit the name and the description of your batch search if needed:

1. You can choose to delete the current batch search after relaunching it:

Note: if you're worried about losing your previous results because of an error, we recommend to keep your current batch search (turn off this toggle button) and delete it only after the relaunch is a success.

1. Click 'Submit':

You can see your relaunched batch search running in the batch search's list:

I get a "failure". What does that mean?

Failures in batch searches can be due to several causes.

Click the 'See error' button to open the error window:

The first query containing an error makes the batch search fail and stop.

Check this first failure-generating query in the error window:

We recommend to remove the slash, as well as any reserved characters, and re-run the batch search again.

'elasticsearch: Name does not resolve'

If you have a message which contain 'elasticsearch: Name does not resolve', it means that Datashare can't make Elastic Search, its search engine, work.

Example of a message regarding a problem with ElasticSearch:

SearchException: query='lovelace' message='org.icij.datashare.batch.SearchException: java.io.IOException: elasticsearch: Name does not resolve'

__

'Data too large'

One of your queries can lead to a 'Data too large' error.

It means that this query had too many results or in their results, some documents that were too big to process for Datashare. This makes the search engine fail.

We recommend to remove the query responsible for the error and re-start your batch search without the query which led to the 'Data too large' error.

****

'SearchException: query='AND ada' '

One or several of your queries contains syntax errors.

It means that you wrote one or more of your queries the wrong way with some characters that are reserved as operators (see below).

Datashare stops at the first syntax error. It reports only the first ​error. You might need to check all your queries as some errors can remain after correcting the first one.

They are more likely to happen when 'do phrase matches' toggle button is turned off:

When 'Do phrase matches' is on, syntax error can still happen though:

Here are the most common errors:

- A query starts with AND (all uppercase)

- A query starts with OR (all uppercase)

- A query contains only one double quote or a double quote in a word

- A query starts with or contains tilde (~) inside a term

- A query starts with or contains a caret (^)

- A query contains one slash (/)

- A query uses square brackets ([ ])

Delete your batch search

Open your batch search and click the trash icon:

Then click 'Yes':