What is Datashare?

Welcome to Datashare - a self-hosted documents search software. It is a free and open-source software developed by the International Consortium of Investigative Journalists (ICIJ). Initially created to combine multiple named-entity recognition pipelines, this tool is now a fully-featured search interface to dig into your documents.

With the help of several open-source tools (Extract, , , , , , and more), Datashare can be used on one single personal computer, as well as on 100 interconnected servers.

Who uses it?

Datashare is developed by the ICIJ, a collective of investigative journalists. Datashare is built at the top of technologies and methods already tested with investigations like the or the .

Seeing the growing interest for ICIJ's technology, we decided to open source this key component of our investigations so a single journalist as well as big media organizations could use it for their own documents.

Datashare is free so anyone can use it and find is useful.

Curious to know more about how we use Datashare?

Where can I see Datashare in action?

We setup a with a small set of documents from the investigation (2014). When using this instance, you will be assigned a temporary user which can star, tag, recommend and explore documents.

Can I run Datashare on my server?

Datashare was also built to run on a server. This is how we use it for our collaborative projects. Please refer to to know how it works.

Can I customize Datashare?

When building Datashare, one of our first decisions was to use to create an index of documents. It would be fair to describe Datashare as a nice looking web interface for Elasticsearch. We want our search platform to be user-friendly while keeping all the powerful Elasticsearch features available for advanced users. This way we ensure that Datashare is usable by non tech-savvy reporters, but still robust enough to satisfy data analysts and developers who want to query the index directly .

We implemented the possibility to create plugins, to make this process more accessible. Instead of modifying Datashare directly, you could isolate your code with a specific set of features and then configure Datashare to use it. Each Datashare user can pick the plugins they need or want, and have a fully customized installation of our search platform. Please have a look at the .

In which languages is Datashare available?

This project is currently available in English, French and Spanish. You can help improve and complete translations on .

In local mode, Datashare provides a self-contained software application that users can install and run on their own local machines.

The software allows users to search into their documents within their own local environment, without relying on external servers or cloud infrastructure.

This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.

To report a bug, please post an issue in our GitHub detailing your logs with:

• Your Operating System (Mac, Windows or Linux)

• The version of your Operating System

• The version of Datashare

• Screenshots of your issue

• A description of your issue

If, for confidentiality reasons, you don't want to open an issue on Github, please write to .

Once Datashare is installed, go to 'Finder' > 'Applications', and double-click on 'Datashare':

A Terminal window called 'Datashare.command' opens and describes the technical operations going on during the opening:

⇒ Important: Keep this Terminal window open as long as you use Datashare.

Once the process is done, Datashare should now automatically open in your default internet browser. If it doesn’t, type 'localhost:8080' as a URL in your browser.

Datashare must be accessed from your internet browser (Firefox, Chome, etc), even though it works offline without Internet connection (see FAQ: ).

You can now .

The CLI stages are primarily intented to be run for an instance of Datashare that uses non-embedded resources (ElasticSearch, database, key/value memory store). This allows you to distribute heavy tasks between servers.

1. SCAN

This is the first step to add documents to Datashare from the command-line. The SCAN stage allows you to queue all the files that need to be indexed (next step). Once this task is done, you can move to the next step. This stage cannot be distributed.

datashare --mode CLI \  
  # Select the SCAN stage
  --stage SCAN \
  # Where the document are located
  --dataDir /path/to/documents \
  # Store the queued files in Redis
  --dataBusType REDIS \
  # URI of Redis 
  --redisAddress redis://redis:6379

2. INDEX

The INDEX stage is probably the most important (and heavy!) one. It pulls documents to index from the queue created in the previous step, then use a combination of and to extract text, metadata and OCR images. The result documents are stored in ElasticSearch. The queue used to store documents to index is a "blocking list", meaning that only one client can pull a concurrent value at the time. This allows users to distribute this command on several servers.

3. NLP

Once a document is available for search (stored in ElasticSearch), you can use the NLP stage to extract named entities from the text. This process will not only create named entity mentions in ElasticSearch, it will also mark every analyzed document with the corresponding NLP pipeline (CORENLP by default). In other words, the process is idempotent and can be parallelized as well on several servers.

Web modes

There are two modes:

In local mode and embedded mode, Datashare provides a self-contained software application that users can install and run on their own local machines. The software allows users to search into their documents within their own local environments, without relying on external servers or cloud infrastructure. This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.

In server mode, Datashare operates as a centralized server-based system. Users can access to the platform through a web interface, and the documents are stored and processed on Datashare's servers. This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.

Comparaison between modes

The running modes offer advantages and limitations. This matrix summarizes the differences:

When running Datashare in local mode, users can choose to use embedded services (like ElasticSearch, SQLITE, in-memory key/value store) on the same JVM than Datashare. This variant of the local mode is called "embedded mode" and allows user to run Datashare without having to setup any additional software. The embedded mode is used by default.

CLI mode

In cli mode, Datashare starts without a web server and allows user to perform tasks over their documents. This mode can be used in conjunction with both local and server modes, while allowing users to distribute heavy tasks between several servers.

If you want to learn more about which tasks you can execute in this mode, checkout the .

Daemon modes

Those modes are intended to be used for action that requires to "wait" for pendings tasks.

In batch download mode, the daemon waits for a user to request a batch download of documents. When a request is received, the daemon starts a task to download the document matching the user search, and bundle them into a zip file.

In batch search mode, the daemon waits for a user to request a batch search of documents. To create a batch search, users must go through the dedicated form on Datashare where they can upload a list of search terms (in CSV format). The daemon will then start a task to search all matching documents and store every occurrences in the database.

How to change modes

Datashare is shipped as a single executable, with all modes available. As previously mentioned, the default mode is the embedded mode. Yet when starting Datashare in command line, you can explicitly specify the running mode. For instance on Ubuntu/Debian:

The installer will take care of checking that your system have all the dependencies to run Datashare. Because this software use Apache Tesseract (to perform Optical Character Recognition, OCR) and Mac doesn't support them out-of-box, heavy dependencies must be downloaded. If your system have none of those dependencies, the first installation of Datashare can take up to 30 minutes.

The installer will set up:

• Xcode Command Line Tools (if neither XCode or Xcode Command Line Tools are installed)

• Homebrew (if neither Homebrew or MacPorts are installed)

• Apache Tesseract with MacPorts or Homebrew

• Java JRE 17

• Datashare executable

Note: Previous versions of this document referred to a "Docker Installer". We do not provide this installer anymore but Datashare is still and supported with Docker.

Installation fails:

• Error while installing Homebrew or MacPorts: you can manually install first and then restart the installer.

• "System Software from application was blocked from loading" : Check in your Mac's "System Settings" > "privacy & security" if you have a section with this mention "System software from application 'Datashare' was blocked from loading" or something similar related to Datashare. If you have this section you'll have to click "Allow" to be able to install datashare.

• For any other issue check our or with your setup (macOs version) and installer logs (Command+L when the installer is launched and failed).

You can now .

Open the Windows main menu at the left of the bar at the bottom of your computer screen and click on 'Datashare'. (The numbers after 'Datashare' just indicate which version of Datashare you installed.)

A window called 'Terminal' will have opened, showing the progress of opening Datashare. Do not close this black window as long as you use Datashare.

Keep this Terminal window open as long as you use Datashare.

Datashare should now automatically open in your default internet browser. If it doesn’t, type 'localhost:8080' in your browser.

Datashare must be accessed from your internet browser (Firefox, Chome, etc), even though it works offline without Internet connection (see FAQ: ).

You can now .

Prerequisites

Datashare platform is designed to function effectively by utilizing a combination of various services. To streamline the development and deployment workflows, Datashare relies on the use of Docker containers. Docker provides a lightweight and efficient way to package and distribute software applications, making it easier to manage dependencies and ensure consistency across different environments.

Currently, only a .deb package for Debian/Ubuntu is provided.

If you want to run it with another Linux distribution, you can download the latest version of the Datashare jar here:

And adapt the following launch script to your environment: .

Start Datashare by launching it from the command-line:

Datashare should now automatically open in your default internet browser. If it doesn’t, type '' in your browser.

Datashare must be accessed from your internet browser (Firefox, Chome, etc), even though it works offline without Internet connection (see: ).

It's now time to .

Plugins are front-end modules to add new features in Datashare's user interface.

Extensions are back-end modules to add new features to store and manipulate data with Datashare.

Add plugins to Datashare's front-end

To be able to perform OCR, Datashare uses an open source technology called Apache Tesseract. When Tesseract extracts text from images, it uses 'language packages' especially trained for each specific languages. Unfortunately, those packages can be heavy and to ensure a lightweight installation of Datashare, the installer doesn't use them all by default. In the case Datashare informs you of a missing package, this guide explains you how to manually install it on your system.

Install packages on Linux

To add OCR languages on Linux, simply use the following command:

Where `[lang]` is can be :

• all if you want to install all languages

• a language code (ex: fra, for French), the list of languages is available

Install packages on Mac

The Datashare Installer for Mac checks for the existence of either or , which package managers are used for the installation of Tesseract. If none of those two package managers is present, the Datashare Installer will install MacPorts by default.

With MacPorts (default)

First, you must check that MacPort is installed on your computer. Please run in a Terminal:

You should see an output similar to this:

If you get a command not found: port, this either means you are using Homebrew (see next section) or you did not yet.

If MacPort is installed on your computer, you should be able to add the missing Tesseract language package with the following command (for German):

The full list of supported language packages can be found .

Once the installation is done, close and restart Datashare to be able to use the newly installed packages.

With Homebrew

If Homebrew was already present on your system when Datashare was installed, Datashare used it to install Tesseract and its language packages. Because Homebrew doesn't package each Tesseract language individually, all languages are already supported by your system. In other words, you have nothing to do!

If you want to check if Homebrew is installed, run the following command in a Terminal:

You should see an output similar to this:

If you get a command not found: brew error, this mean Homebrew is not installed on your system. You might either use MacPorts (see previous section) or on your computer.

Install languages on Windows

Languages packages are available on Tesseract . Trained data files have to be downloaded and added into tessdata folder in Tesseract's installation folder.

*Additional languages can be also added during Tesseract's installation.

The list of installed languages can be checked with Windows command prompt or Powershell with the command tesseract --list-langs.

Datashare has to be restarted after the language installation. Check how for , and .

Prerequisites

Get Neo4j up and running

Follow the instructions of the to get Neo4j up and running.

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the 'Other platforms and versions' button when downloading to access versions if necessary.

Add entities

If it's not done yet and extract names of people, organizations and locations as well as email addresses.

If your project contains emails, make sure to also extract email addresses.

Next step

You can now .

Install the Neo4j plugin

Install the Neo4j plugin following these instructions.

Configure the Neo4j plugin

1. At the bottom of the menu, click on the 'Settings' icon:

2. Make sure the following settings are properly set:

• Neo4j Host should be localhost or the address where your Neo4j instance is running

• Neo4j Port should be the port where your Neo4j instance is running (7687 by default)

3. When running Neo4j Community Edition, set the Neo4j Single Project value. In community edition, the Neo4j DBMS is restricted to a single database. Since Datashare supports multiple projects, you must set the Neo4j Single Project with the name of the project which will use Neo4j plugin. Other projects won't be able to use the Neo4j plugin.

4. Restart Datashare to apply the changes. Check how for , or .

5. Go to 'Projects' > your project's page > the Graph tab. You should see the Neo4j widget. After a little while, its status should be RUNNING:

You can now .

Create the graph

1. Go to 'All projects' and click on your project's name:

1. Go to the Graph tab and in the first step 'Import', click on the 'Import' button:

You will then see a new import task running.

When the graph creation is complete, 'Graph statistics' will reflect the number of documents and entities nodes found in the graph:

Update the graph

If new documents or entities are added or modified in Datashare, you will need to update the Neo4j graph to reflect these changes.

Go to 'All projects' > one project's page > the 'Graph' tab. In the first step, click on the 'Update graph' button:

To detect whether a graph update is needed, go to the 'Projects' page and open your project:

Compare the number of documents and entities found in Datashare in 'Projects' > 'Your project' > 'Insights'...

...with the numbers found in your project in the 'Graph' tab. Run an update in case of mismatch:

The update will always add missing nodes and relationships, update existing ones if they were modified, but will never delete graph nodes or relationships.

You can now using your favorite visualization tool.

This document assumes that you have installed Datashare in server mode within Docker and already added documents to Datashare.

In server mode, it's important to understand that Datashare does not provide an interface to add documents. As there is no build-in roles and permission in Datashare's data model, we have no way to differentiate users to offer admin additional tools.

This is likely to be changed in the near future, but in the meantime, you can extract named entities using the command-line interface.

Datashare has the ability to detect email addresses, name of people, organizations and locations. This process use a Natural Language Processing (NLP) pipeline called CORENLP. Once your documents have been indexed in Datashare, you can perform the named entities extraction in the same fashion as the previous CLI's stages:

What's happening here:

• Datashare starts in "CLI"

• We ask to process the NLP

• We tell Datashare to use the elasticsearch service

Datashare will use the output queue from the previous INDEX stage (by default extract:queue:nlp in Redis) that contains all the document ids to be analyzed.

The first time you run this command you will have to wait a little bit because Datashare need to download CORENLP's models which can be big.

You can also use chain the 3 stages altogether:

As for the previous you may want to restore the output queue from the INDEX stage. You can do:

The added ENQUEUEIDX stage will read Elasticsearch index, find all documents that have not already been analyzed by the CORENLP NER pipeline, and put the IDs of those documents into the extract:queue:nlp queue.

Prerequisites

Datashare platform is designed to function effectively by utilizing a combination of various services. To streamline the development and deployment workflows, Datashare relies on the use of Docker containers. Docker provides a lightweight and efficient way to package and distribute software applications, making it easier to manage dependencies and ensure consistency across different environments.

Read more about how to install Docker on your system.

Starting Datashare with multiple containers

Within Datashare, Docker Compose can play a significant role in enabling the setup of separated and persistent services for essential components. By utilizing Docker Compose, you can define and manage multiple containers as part of a unified service. This allows for seamless orchestration and deployment of interconnected services, each serving a specific purpose within the Datashare ecosystem.

Specifically, Docker Compose allows you to configure and launch separate containers for PostgreSQL, Elasticsearch, and Redis. These containers can be set up in a way that ensures their data is persistently stored, meaning that any information or changes made to the database, search index, or key-value store will be retained even if the containers are restarted or redeployed.

This separation of services using Docker Compose provides several advantages. It enhances modularity, scalability, and maintainability within the Datashare platform. It allows for independent management and scaling of each service, facilitating efficient resource utilization and enabling seamless upgrades or replacements of individual components as needed.

To start Datashare in server mode with , you can use the following docker-compose.yml file for version 20.1.4 (check latest version on ):

Open a terminal or command prompt and navigate to the directory where you saved the docker-compose.yml file. Then run the following command to start the Datashare service:

The -d flag runs the containers in detached mode, allowing them to run in the background.

Docker Compose will pull the necessary Docker images (if not already present) and start the containers defined in the YAML file. Datashare will take a few seconds to start. You can check the progression of this opperation with:

Once the containers are up and running, you can access the Datashare service by opening a web browser and entering http://localhost:8080. This assumes that the default port mapping of 8080:8080 is used for the Datashare container in the YAML file.

To stop the Datashare service and remove the containers, you can run the following command in the same directory where the docker-compose.yml file is located:

This will stop and remove the containers, freeing up system resources.

Add documents to Datashare

If you reach that point, Datashare is up and running but you will discover very quickly that no documents is available in the search results. Next step: .

Extract named entities

Datashare has the ability to detect email addresses, name of people, organizations and locations. You must perform the named entities extraction in the same fashion than the previous commands. Final step: .

Authentication with Datashare in server mode is the most impacting choice that has to be made. It can be one of the followings:

• Basic authentication with credentials stored in database (PostgreSQL)

• Basic authentication with credentials stored in Redis

• OAuth2 with credentials provided by an identity provider (KeyCloak for example)

• Dummy basic auth to accept any user (⚠️ if the service is exposed to internet, it will leak your documents)

This document assumes that you have installed Datashare in server mode within Docker.

In server mode, it's important to understand that Datashare does not provide an interface to add documents. As there is no build-in roles and permission in Datashare's data model, we have no way to differentiate users to offer admin additional tools.

This is likely to be changed in the near future, but in the meantime, you can still add documents to Datashare using the command-line interface.

Here is a simple command to scan a directory and index its files:

What's happening here:

• Datashare starts in "CLI"

• We ask to process both SCAN and INDEX at the same time

• The SCAN stage feeds a queue in memory with file to add

• The INDEX stage pulls files from the queue to add them to ElasticSearch

• We tell Datashare to use the elasticsearch service

• Files to add are located in /home/datashare/Datashare/ which is a directory mounted from the host machine

Alternatively, you can do this in two separated phases, as long as you tell Datashare to store the queue in a shared resource. Here, we use the Redis:

Once the operation is done, we can easily check the content of queue created by Datashare in Redis. In this example we only display the 20 first files in the datashare:queue:

The INDEX can now be executed in the same container:

Once the indexing is done, Datashare will exit gracefully and your document will already be visible on Datashare.

Sometimes you will face the case where you have an existing index, and you want to index additional documents inside your working directory without processing every document again. It can be done in two steps :

• Scan the existing ElasticSearch index and gather document paths to store it inside a report queue

• Scan and index (with OCR) the documents in the directory, thanks to the previous report queue, it will skip the paths inside of it

You can have a dummy authentication that always accepts basic auth. So you should see this popup:

But then whatever user or password you type, it will enter Datashare.

Example

docker run -ti ICIJ/datashare -m SERVER \
  --dataDir /home/dev/data \
    --batchQueueType REDIS \
    --dataSourceUrl 'jdbc:postgresql://postgres/datashare?user=dstest&password=test'\
    --sessionStoreType REDIS \
    --authFilter org.icij.datashare.session.YesBasicAuthFilter

Basic authentication is a simple protocol that uses the HTTP headers and the browser to authenticate users. User credentials are sent to the server in the header Authorization with user:password base64 encoded:

Authorization: Basic dXNlcjpwYXNzd29yZA==

It is secure as long as the communication to the server is encrypted (with SSL for example).

On the server side, you have to provide a database user inventory. You can launch datashare first with the full database URL, then Datashare will automatically migrate your database schema. Datashare supports SQLite and PostgreSQL as back-end databases. SQLite is not recommended for a multi-user server because it cannot be multithreaded, so it will introduce contention on users' DB SQL requests.

Then you have to provision users. The passwords are sha256 hex encoded (for example with bash):

$ echo -n bar | sha256sum
fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9  -

Then you can insert the user like this in your database:

If you use other indices, you'll have to include them in the group_by_applications, but local-datashare should remain. For example if you use myindex:

Or you can use COPY statement if you want to create them all at once.

Then when accessing Datashare, you should see this popup:

Example

Here is an example of launching Datashare with Docker and the basic auth provider filter backed in database:

Install the Neo4j plugin

Install the Neo4j plugin using the Datashare CLI so that users can access it from the frontend:

Installing the plugin installs the datashare-plugin-neo4j-graph-widget plugin inside /home/datashare/plugings and will also install the datashare-extension-neo4j backend extension inside /home/datashare/extensions. These locations can be changed by updating the docker-compose.yml.

Configure the Neo4j extension

Update the docker-compose.yml to reflect your .

If your choose a different Neo4j user or set a password for your Neo4j user make sure to also set DS_DOCKER_NEO4J_USER and DS_DOCKER_NEO4J_PASSWORD.

When running Neo4j Community Edition, set the DS_DOCKER_NEO4J_SINGLE_PROJECT value. In community edition, the Neo4j DBMS is restricted to a single database. Since Datashare supports multiple projects, you must set the DS_DOCKER_NEO4J_SINGLE_PROJECT with the name of the project which will use Neo4j plugin. Other projects won't be able to use the Neo4j plugin.

Restart Datasahre

After installing the plugin a restart might be needed for the plugin to display:

Next step

You can now .

In server mode, Datashare operates as a centralized server-based system. Users can access the platform through a web interface, and the documents are stored and processed on Datashare's servers.

This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.

Launch configuration

Datashare is launched with --mode SERVER and you have to provide:

• The external elasticsearch index address elasticsearchAddress

• A Redis store address redisAddress

• A Redis data bus address messageBusAddress

Example:

This is the default authentication mode: if not provided in CLI, it will be selected. With OAuth2 you will need a third-party authorization service. The diagram below describes the workflow:

Example

docker run -ti ICIJ/datashare:version --mode SERVER \
    --oauthClientId 30045255030c6740ce4c95c \
    --oauthClientSecret 10af3d46399a8143179271e6b726aaf63f20604092106 \
    --oauthAuthorizeUrl https://my.oauth-server.org/oauth/authorize \
    --oauthTokenUrl https://my.oauth-server.org/oauth/token \
    --oauthApiUrl https://my.oauth-server.org/api/v1/me.json \
    --oauthCallbackPath /auth/callback

Integration with KeyCloak

We made a small demo to show how it could be setup.

Run the Neo4j extension CLI

The Neo4j related features are added to the DatashareCLI through the extension mechanism. In order to run the extended CLI, the Java CLASSPATH must be extended with the path of the datashare-extension-neo4j jar. By default, this jar is located in /home/.local/share/datashare/extensions/*, so the CLI will be run as following:

Create the graph

In order to create the graph, run the --fullImport command for your project:

The CLI will display the import task progress and log import related information.

Update the graph

When new documents or entities are added or modified inside Datashare, you will need to update the Neo4j graph to reflect these changes.

To update the graph, you can just re-run the full export:

The update will always add missing nodes and relationships, update existing ones if they were modified, but will never delete graph nodes or relationships.

To detect whether a graph update is needed, go to the 'Projects' page and open your project:

Compare the number of documents and entities found in Datashare in 'Projects' > 'Your project' > 'Insights'...

...with the numbers found in your project in the 'Graph' tab. Run an update in case of mismatch:

The update will always add missing nodes and relationships, update existing ones if they were modified, but will never delete graph nodes or relationships.

You can now using your favorite visualization tool.

Improving the performance of Datashare involves several techniques and configurations to ensure efficient data processing. Extracting text from multiple file types and images is an heavy process so be aware that even if we take care of getting the best performances possible with Apache Tika and Tesseract OCR, this process can be expensive. Below are some tips to enhance the speed and performance of your Datashare setup.

Separate Processing Stages

Execute the SCAN and INDEX stages independently to optimize resource allocation and efficiency.

Examples:

Distribute the INDEX Stage

Distribute the INDEX stage across multiple servers to handle the workload efficiently. We often use multiple instances (32 CPUs, 128 GB of memory) with a remote Redis and a remote ElasticSearch instance to alleviate processing load.

For projects like the (2.94 TB), we ran the INDEX stage to up to 10 servers at the same time which cost ICIJ several thousand of dollars.\

Leverage Parallelism

Datashare offers --parallelism and --parserParallelism options to enhance processing speed.

Example (for g4dn.8xlarge with 32 CPUs):

Optimize ElasticSearch

ElasticSearch can significantly consume CPU and memory, potentially becoming a bottleneck. For production instance of Datashare, we recommend deploying ElasticSearch on a remote server to improve performances.

Adjust JAVA_OPTS

You can fine-tune the JAVA_OPTS environment variable based on your system's configuration to optimize Java Virtual Machine memory usage. &#xNAN;Example (for g4dn.8xlarge8with 120 GB Memory):

Specify Document Language

If the document language is known, explicitly setting it can save processing time.

• Use --language for general language setting (e.g., FRENCH, ENGLISH).

• Use --ocrLanguage for OCR tasks to specify the Tesseract model (e.g., fra, eng).

Example:

Manage OCR Tasks Wisely

OCR tasks are resource-intensive. If not needed, disabling OCR can significantly improve processing speed. You can disable OCR with --ocr false.

Example:

Efficient Handling of Large Files

Large PST files or archives can hinder processing efficiency. We recommend extracting these files before processing with Datashare. If they are too many of them, keep in mind that Datashare will be able to extract them anyway.

Example of splitting Outlook PST files in multiple .eml files with :

Basic authentication is a simple protocol that uses the HTTP headers and the browser to authenticate users. User credentials are sent to the server in the header Authorization with user:password base64 encoded:

Authorization: Basic dXNlcjpwYXNzd29yZA==

It is secure as long as the communication to the server is encrypted (with SSL for example).

On the server side, you have to provide a user store for Datashare. For now we are using a Redis data store.

So you have to provision users. The passwords are sha256 hex encoded. For example using bash:

Then insert the user like this in Redis:

If you use other indices, you'll have to include them in the group_by_applications, but local-datashare should remain. For exammple if you use myindex:

Then you should see this popup:

Example

Here is an example of launching Datashare with Docker and the basic auth provider filter backed in Redis:

Double quotes for exact phrase

To have all documents mentioning an exact phrase, you can use double quotes. Use straight double quotes ("example"), not curly double quotes (“example”).

"Alicia Martinez’s bank account in Portugal"

OR (or space)

To have all documents mentioning at least one of the queried terms, you can use a simple space between your queries (as OR is the default operator in Datashare) or OR . You need to write OR with all letters uppercase.

Alicia Martinez

Alicia OR Martinez

AND (or +)

To have all documents mentioning all the queried terms, you can use AND between your queried words. You need to write AND with all letters uppercase.

Alicia AND Martinez

+Alicia +Martinez

NOT (or ! or -)

To have all documents NOT mentioning some queried terms, you can use NOT before each word you don't want. You need to write NOT with all letters uppercase.

NOT Martinez

!Martinez

-Martinez

Combine operators

Parentheses should be used whenever multiple operators are used together and you want to give priority to some.

((Alicia AND Martinez) OR (Delaware AND Pekin) OR Grey) AND NOT "parking lot"

You can also combine these with regular expressions (regex) between two slashes ().

Wildcards

If you search faithf?l, the search engine will look for all words with all possible single character between the second f and the l in this word. It also works with * to replace multiple characters.

Alicia Martin?z

Alicia Mar*z

Fuzziness

You can set fuzziness to 1 or 2. It corresponds to the maximum number of operations (insertions, deletions, substitutions and transpositions) on characters needed to make one term match the other.

kitten -> sitten (1 substitution (k turned into s) = fuzziness is 1)

kitten -> sittin (2 substitutions (k turned into s and e turned into i) = fuzziness is 2)

If you search for similar terms (to catch typos for example), you can use fuzziness. Use the at the end of the word to set the fuzziness to 1 or 2.

"The default edit distance is 2, but an edit distance of 1 should be sufficient to catch 80% of all human misspellings. It can be specified as: quikc~1" (source: ).

quikc~ brwn~ foks~ (as the default edit distance is 2, this query will catch all quick, quack, quock, uqikc, etc. as well as brown, folks, etc.)

Datashare~1 (this query will catch Datasahre, Dqtashare, etc.)

Proximity searches

When you type an exact phrase (in double quotes) and use fuzziness, then the meaning of the fuzziness changes. Now, the fuzziness means the maximum number of operations (insertions, deletions, substitutions and transpositions) on terms needed to make one phrase match the other.

Examples:

"the cat is blue" -> "the small cat is blue" (1 insertion = fuzziness is 1)

"the cat is blue" -> "the small is cat blue" (1 insertion + 2 transpositions = fuzziness is 3)

"While a phrase query (eg "john smith") expects all of the terms in exactly the same order, a proximity query allows the specified words to be further apart or in a different order. A proximity search allows us to specify a maximum edit distance of words in a phrase." (source: ).

"fox quick"~5 (this query will catch "quick brown fox", "quick brown car thin fox" or even "quick brown car thin blue tree fox"

The closer the text in a field is to the original order specified in the query string, the more relevant that document is considered to be. When compared to the above example query, the phrase quick fox would be considered more relevant than quick brown fox(source: ).

Boosting operators

Use the boost operator ^ to make one term more relevant than another. For instance, if we want to find all documents about foxes, but we are especially interested in quick foxes:

quick^2 fox

The default boost value is 1, but can be any positive floating point number. Boosts between 0 and 1 reduce relevance. Boosts can also be applied to phrases or to groups:

"john smith"^2 (foo bar)^4

(source: )

Regular expressions (Regex)

‌"A regular expression (shortened as regex or regexp) is a sequence of characters that define a search pattern." ().

1. You can use Regex in Datashare. Regular expressions (Regex) in Datashare need to be written between 2 slashes and starting with the field (content, name, author, recipients, etc):

content: /.*..*@.*..*/

The example above will search in the content of the document for any expression which is structured like an email address with a dot between two expressions before the @ and a dot between two expressions after the @ like in 'first.lastname@email.com' for instance.

2. Regex can be combined with standard queries in Datashare :

("Ada Lovelace" OR "Ado Lavelace") AND paris AND content:/.*..*@.*..*/

3. You need to escape the following characters by typing a backslash just before them (without space):‌ . ? + * | { } [ ] ( ) " \ # @ & < > ~

/.*..*\@.*..*/ (the @ was escaped by a backslash \ just before it)

4. Important: Datashare relies on Elastic's Regex syntax as explained. Datashare uses . A consequence of this is that spaces cannot be searched as such in Regex.

We encourage you to use the AND operator to work around this limitation and make sure you can make your search.

If you're looking for French International Bank Account Number (IBAN) that can or cannot contain space and contain FR followed by numbers and/or letters (it could be FR7630001007941234567890185 ou FR76 3000 4000 0312 3456 7890 H43 for example), you can then search for:

/FR[0-9]{14}[0-9a-zA-Z]{11}/ OR (/FR[0-9]{2}.*/ AND /[0-9]{4}.*/ AND /[0-9a-zA-Z]{11}.*/)

Here are a few examples of useful Regex:

• You can search for /Dimitr[iyu]/ instead of searching for Dimitri OR Dimitry OR Dimitru. It will find all the Dimitri, Dimitry or Dimitru.

• You can search for /Dimitr[^yu]/ if you want to search all the words which begin with Dimitr and do not end with either y nor u.

Other common Regex examples:

• phone numbers with "-" and/or country code like +919367788755, 8989829304, +16308520397 or 786-307-3615 for instance: /[\+]?[(]?[0-9]{3}[)]?[-\s.]?[0-9]{3}[-\s.]?[0-9]{4,6}/

• emails (): /[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+.[a-zA-Z0-9-.]+/

You can find many other examples . More generally, if you use a regex found on internet, beware that the syntax is not necessarily compatible with elasticsearch's. For example \d, \S and the like .

Search with metadata fields

Prerequisites

Get Neo4j up and running

Follow the instructions of the to get Neo4j up and running.

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the 'All platforms and versions' button when downloading to access versions if necessary.

Add entities

If it's not done yet add entities to your project .

If your project contains email documents, make sure to run the EMAIL pipeline together with regular NLP pipeline. To do so add set the follow nlpp flag to --nlpp CORENLP,EMAIL.

Next step

You can now .

Expand the menu to go to 'Projects' > 'All projects':

Search in projects' names using the search bar on the right:

Sort your projects by clicking the top right Settings icon:

In the Page settings, choose a sort by option, change the number of projects per page or the layout:

To explore a project, close the Settings and click on the name of the project:

You can now .

Filters

Open 'Filters' on the left of the search bar:

'Indexing dates' arethe dates when the documents were added to Datashare.

'Extraction levels' regard embedded documents:

• The 'file on disk' is level zero

• If a document is attached to (or contained in) a file on disk, its extraction level is '1st'

• If a document is attached to (or contained in) a document itself contained in a file on disk, its extraction level is '2nd'

Filter by entities

If you asked Datashare to 'Find entities' and the task was complete, you will see names of people, organizations, locations and e-mail adresses in the filters. These are the entities automatically detected by Datashare:

Exclude filters

Tick the 'Exclude' checkbox to select all items except those selected.

In the search breadcrumb, you see that the excluded filters are strikethrough:

Contextualize filters

In most filters, tick 'Contextualize' to update the number of documents indicated in the filters so they reflect the results.

The filter will only count what you selected, it will reflect the results of your current selection:

Clear all filters

To reset all filters at the same time, open the search breadcrumb:

Click 'Clear filters':

See a document in full-screen view

Open a document in 'Search' > 'Documents' > one document and click the icon with in and out arrows (this applies to the List layout while in Grid and Table layout, the documents always open in full-screen view):

You now see the document in full screen view and can go to the next document in your results by using the pagination carousel on the top of the screen:

Search in a document

• Open a document in 'Search' > 'Documents' > one document

• Stay on the first tab called 'Text'. This tab shows the text as extracted from your document by Datashare.

• Click on the search bar or press Command (⌘) / Control + F

See original document

Go to the 'View' tab to see the original document.

Note: this visualization of the document is available only for some file types: images, PDF, CSV, xlsx and tiff but not other file types like Word documents or e-mails for instance.

Search for attachments and documents in the same folder

Go to the 'Metadata' tab and click on 'X documents in the same folder' or 'Y children documents':

You see the list of documents. To open all the documents in the same folder or all the children documents, click 'Search all' below. There is no 'Search all' button if there is no documents, as for the children documents below:

Explore metadata

Go the 'Metadata' tab to explore all the properties of the document:

If a metadata is interesting to you and you'd like to know if other documents in your project share the same metadata, click the search icon:

You can also copy or pin a metadata.

Entities

In the 'Entities' tab, only if you previously run tasks to in Datashare, you read the name of people, organizations, locations and e-mail adresses, along with the number of their occurrences in the document:

Hover one entity to see a popover with all their mentions in context in the document by clicking on the arrows:

Go to the 'Info' tab to check how the entity was extracted:

Expand the menu, open 'All projects' and click on the name of the project that you want to explore:

If you'd like to pin this project in the menu for an easy access, click 'Pin to menu':

Your project is now pinned in the menu:

In a project page, in the first tab called 'Insights', you find statistics and a bar chart displaying the number of documents by creation date.

Filter this chart by path by clicking 'Select path':

Click on one bar for a year or month to see all the corresponding documents:

On the 'Languages', 'File Types' and 'Authors' widgets, you see stats:

Search all documents with a specific criteria, for instance here with the French language:

Finally, in the server collaborative mode, you see the Latest recommended documents, that is to say the documents marked as recommended by other members of the project:

You can now .

You must have added documents in Datashare before. Check how for Mac, Windows and Linux.

Search bar

Expand the menu to go to 'Search' > 'Documents':

Make room by closing the menu:

Type terms in the search bar and press Enter:

Default operator is OR

If you type several terms separated by space, as the default operator is OR, Datashare will search for all documents containing at least one of the searched terms.

For instance, Datashare finds documents containing either 'ikea' or 'paris' or both terms here:

Linked entities

As you type a term, Datashare suggest linked entities - only if a task to find entities in this project was completed.

Press Esc on your keyboard to close the dropdown or click on one of the entities to replace your term in the search bar:

Search in a field

Search within a specific field only, by using the dropdown 'All fields':

Search breadcrumb

To see your queries in the search breadcrumb, click on the icon on the left of the search bar:

If you'd like to remove all searched terms from the search bar, click 'Clear query':

Results settings

To change the page settings, click the Settings icon on the top right:

You can change Sort by, Documents per page, Layout and also Properties:

Ticking these properties will change which document's metadata are displayed in the results, in the document cards, in all 3 layouts (List, Grid, Table):

You can now make your search more precise .

Warning: this requires some technological knowledge.

You can make Datashare follow soft links : add --followSymlinks when Datashare is launched.

If you're on Mac or Windows, you must change the launch script.

If you're on Linux, you can add the option after the Datashare command.

Datashare was created with scalability in mind which gave ICIJ the ability to index terabytes of documents.

To do so, we used a cluster of dozens of EC2 instances on AWS, running on Ubuntu 16.04 and 18.04. We used c4.8xlarge instances (36 CPUs / 60 GB RAM).

The most complex operation is OCR (we use Apache Tesseract) so if your documents don't contain many images, it might be worth deactivating it (--ocr false).

You need an internet connection to install Datashare.

You also need the internet to find people, organizations and locations in documents the first time you use any new NLP option because the models which find these named entities are downloaded when you ask for finding named entities the first time. Subsequently, you don't need an internet connection to find named entities after.

You don't need internet connection to:

• Add documents to Datashare

• Find named entities (except for the first time you use an NLP options - see above)

• Search and explore documents

• Download documents

This allows you to work safely on your documents. No third-party should be able to intercept your data and files while you're working offline on your computer.

In local mode, you cannot remove a single document or a selection of documents from Datashare. But you can remove all your projects and documents from Datashare.

Open the menu and on the bottom of the menu, click the trash icon:

A confirmation window opens. The action cannot be undone. It removes all the projects and their documents from Datashare. Click 'Yes' if you are sure:

For advanced users - if you'd like to do it with the Terminal, here are the instructions:

• If you're using Mac: rm -Rf ~/Library/Datashare/index

• If you're using Windows: rd /s /q "%APPDATA%"\Datashare\index

• If you're using Linux: rm -Rf ~/.local/share/datashare/index

Open the menu > 'Search' > 'Documents' and click the keyboard icon at the bottom of the menu:

It opens a window with the shortcuts for your OS (Mac, Windows, Linux):

Click on 'See all shortcuts' to reach the full page view:

Star documents

Star a single document

Click the star icon either at the right of the document's card or at the top right of the document:

Click on the same icons to unstar.

Star multiple documents

Open the selection mode by clicking the multiple cards icon on the left of the pagination:

Select the documents you want to star:

Click the star filled icon:

To unstar documents, click the three-dot icon if necessary and click Unstar:

Filter starred documents

Open the filters by clicking the 'Filters' button on the left of the search bar:

In the 'User data' category, open 'Starred' and tick the 'Starred' checkbox:

Tag documents

Tag a single document

Open a document in 'Search' > 'Documents' > open on a document and above the document's name, click on the hashtag icon:

It opens the Tags panel on the left:

Type your tag and press Enter or click 'Add':

Your tag is now displayed in the 'Added by you' category:

Remove your tag, or others' tags, by clicking their cross icon:

Tag multiple documents

Open the selection mode by clicking the multiple cards icon on the left of the pagination:

Select the documents you want to tag:

Click the three-dot icon if necessary and click 'Tag':

Type your tag or type multiple tags by separating them with comma and click 'Add':

Remove your tag, or others' tags, by clicking their cross icon on each single document (you cannot untag multiple documents):

Filter tagged documents

Open the filters by clicking the 'Filters' button on the left of the search bar:

In the 'User data' category, open 'Tags' and tick the 'Tag' checkboxes for tagged documents you want to filter:

Recommend a document

Open a document in 'Search' > 'Documents' > open on a document and above the document's name, click on the eyes icon:

It opens the Recommendations panel on the left:

Click on the 'Mark as recommended' button:

The document is now marked as recommended by you:

Click 'Unmark as recommended' to unmarked it as recommended.

Filter recommended documents

Open the filters by clicking the 'Filters' button on the left of the search bar:

In the 'User data' category, open 'Recommended by' and tick the 'Username' checkboxes for documents recommended by the users you want to filter:

Yes, you can download a document from Datashare.

Download a document

Open the menu > 'Search' > 'Documents' and click on the download icon on the right of documents' cards:

...or on the top right of an opened document:

Batch download documents

You can also batch download all the documents that match a search. It is limited to 100.00MB.

Open the menu > 'Search' > 'Documents', make queries and apply filter. Once all the results of a specific search are relevant to you, click on the download icon on the right of results:

Find your batch downloads as zip files in the menu > 'Tasks' > 'Batch downloads':

Click on a batch download's name to download it:

Can't download?

If you can't download a document, it means that:

• either Datashare has been badly initialized. Please restart Datashare. If you're an advanced user, you can capture the logs and create an issue on .

• or you are using the server collaborative mode and the admins prevented users from downloading documents

You can use Datashare with multiple users accessing a centralized database on a server.

Warning: to put the server mode in place and to maintain it requires some technical knowledge.

Please find the documentation here.

An entity in Datashare is the name of people, organizations or locations or an email address.

Datashare’s Named Entity Recognition (NER) uses pipelines of Natural Language Processing (NLP), a branch of artificial intelligence, to automatically detect entities in your documents.

You can filter documents by their entities and see all the entities mentioned in a document.

Pipelines of Natural Language Processing are tools that automatically identify entities in your documents. You can only choose one model at a time for one entity detection task.

Open the menu > 'Tasks' > 'Entities' and follow these instructions. Select 'CoreNLP' if you want to use the model with the highest probability of working in most of documents.

In Datashare, for technical reasons, it is not possible to open the 10,000th document.

Example: you search for "Paris", you get 15,634 results. You'd be able to see the first 9,999th results but no more. This also happens if you didn't run any search.

As it is not possible to fix this, here are some tips:

• Refine your search: use filters to narrow down your results and ensure you have less than 10,000 matching documents

• Change the sorting of your results: use 'creation date' or 'alphabetical order' for instance, instead of the sorting by default which corresponds to a relevance scoring

• Search your query in a : you will get all your results either on the batch search results' page or, by downloading your results, in a spreadsheet. From there, you will be able to open and read all your documents

Mac

1. Go to Applications

2. Click right on 'Datashare' and click 'Move to Bin'

Windows

Follow the steps here:

Linux

Use the following command:

sudo apt remove datashare-dist

As a search operator

In the main search bar, you can write a query with the search operator tilde (~) with a number, at the end of each word of your query. You can set fuzziness to 1 or 2. It corresponds to the maximum number of operations (insertions, deletions, substitutions and transpositions) on characters needed to make one term match the other.

kitten -> sitten (1 substitution (k turned into s) = fuzziness is 1)

kitten -> sittin (2 substitutions (k turned into s and e turned into i) = fuzziness is 2)

If you search for similar terms (to catch typos for example), use fuzziness. Use the at the end of the word to set the fuzziness to 1 or 2.

"The default edit distance is 2, but an edit distance of 1 should be sufficient to catch 80% of all human misspellings. It can be specified as: quikc~1" (source: ).

Example: quikc~ brwn~ foks~ (as the default edit distance is 2, this query will catch all quick, quack, quock, uqikc, etc. as well as brown, folks, etc.)

Example: Datashare~1 (this query will catch Datasahre, Dqtashare, etc.)

In batch searches

When you run a , you can set the fuzziness to 0, 1 or 2. It is the same as explained above, it will apply to each word in a query and corresponds to the maximum number of operations (insertions, deletions, substitutions and transpositions) on characters needed to make one term match the other.

kitten -> sitten (1 substitution (k turned into s) = fuzziness is 1)

kitten -> sittin (2 substitutions (k turned into s and e turned into i) = fuzziness is 2)

If you search for similar terms (to catch typos for example), use fuzziness. Use the at the end of the word to set the fuzziness to 1 or 2.

"The default edit distance is 2, but an edit distance of 1 should be sufficient to catch 80% of all human misspellings. It can be specified as: quikc~1" (source: ).

Example: quikc~ brwn~ foks~ (as the default edit distance is 2, this query will catch all quick, quack, quock, uqikc, etc. as well as brown, folks, etc.)

Example: Datashare~1 (this query will catch Datasahre, Dqtashare, etc.)

Prerequisites

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature. To download a specific version, click on 'All platforms and versions' here.

If you are not familiar with graph and Neo4j, take a look at the following resources:

• Find out

• Learn

• Check out

The documents and entities graph

is a graph database technology which lets you represent your data as a graph.

Inside Datashare, Neo4j lets you connect entities between them through documents in which they appear.

After creating a graph from your Datashare project, you will be able to explore this graph and visualize these kinds of relationships between you project entities:

In the above graph, we can see 3 e-mail document nodes in orange, 3 e-mail address nodes in red, 1 person node in green and 1 location node in yellow. Reading the relationship types on the arrows, we can deduce the following information from the graph:

• shapp@caiso.com emailed 20participants@caiso.com, the sent email has an ID starting with f4db344...

• One person named vincent is mentioned inside this email, as well as the california location