• your Operating System (Mac, Windows or Linux)

• the version of your Operating System

• the version of Datashare

• screenshots of your issue

• a description of your issue.

What is Datashare?

Who uses it?

Curious to know more about how we use Datashare?

Where can I see it in action?

Can I run it on my server?

Datashare was also built to run on a server. This is how we use it for our collaborative projects. Please refer to the server documentation to know how it works.

Can I customize Datashare?

Translations

In local mode, Datashare provides a self-contained software application that users can install and run on their own local machines. The software allows users to search into their documents within their own local environments, without relying on external servers or cloud infrastructure. This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.

Apply filters

On the left column, you can apply filters by ticking them, like 'Portable Document Format (PDF)' in File Types and 'English' in Languages in the example below:

A reminder of the currently applied filters, as well as your queried terms, are displayed at the top of the results' column. You can easily unselect these filters from there by clicking them or clear all of them:

The currently available filters are:

• Projects: if you have more than one project, you can select several of them and run searches in multiple projects at once.

• Starred: If you have starred documents, you can easily find them again.

• Tags: If you wrote some tags, you will be able to select and search for them.

• Recommended by: available only on server (collaborative) mode, this functionality helps you find the document recommended by you and/or others.

• File type: This is the 'Content type' of the file (Word, PDF, JEPG image, etc.) as you can read it in a document's 'Tags & Details'.

• Creation dates: the calendar allows you to select a single creation date or a date range. This is when the document was created as it is noticed in their properties. You can find this in a document's 'Tags & Details'.

• Languages: Datashare detects the main language of each document.

• People / Organizations / Locations: you can selected these named entities and search them.

• Path: This is the location of your documents as it is indicated in your original files (ex: desktop/importantdocuments/mypictures). You can find this in a document's 'Tags & Details'.

• Indexing date: This date corresponds to when you indexed the documents in Datashare.

• Extraction level: This regards embedded documents. The file on disk is level zero. If a document (pictures, etc) is attached or contained in a file on disk, extraction level is “1st”. If a document is attached or contained in a document itself contained in a file on disk, extraction level is “2nd”, etc.

Filters can be combined together and combined with searches in order to refine results.

Filter by named entities

If you have asked Datashare to 'Find people, organizations and locations', you can see names of individuals, organizations and locations in the filters. These are the named entities automatically detected by Datashare.

Search for named entities in the filter's search bar:

Select all of them, one or several of them to filter the documents that mention them:

Use the 'Exclude' button

If you want to select all items except one or several of them, you can use the 'Exclude button'.

It allows you to search for all documents which do not correspond to the filter(s) you selected, that is to say to the currently strikethrough filters.

Contextualize filters

In several filters, you can tick 'Contextualize' : this will update the number of documents indicated in the filters in order to reflect the results. The filter will only count what you selected.

In the example below, the 'Contextualize' checkboxes are not ticked:

After the Contextualize button in Tags filter is ticked:

After the Languages button in Tags filter is ticked:

Clear all filters

To reset all filters at the same time, click 'Clear all':

Star a document

• Hover the document's title in the main column. A button with the labe 'Star document' appears.

• Click it:

The starred document is now marked with a star icon:

Unstar a document

Click again on its star icon. The document won't have a star anymore.

Find your starred documents

• Open the 'Starred' filter on the left:

• Tick 'Starred' (respectively 'Unstarred').

• Only starred (respectively unstarred) documents will be displayed in the main column:

The CLI stages are primarly intented to be run for an instance of Datashare that uses non-embedded resources (ElasticSearch, database, key/value memory store). This allows you to distribute heaving tasks between servers.

1. SCAN

This is the first step to add documents to Datashare from the command-line. The SCAN stage allows you to queue all the files that need to be indexed (next step). Once this task is done, you can move to the next step. This stage cannot be distributed.

datashare --mode CLI \  
  # Select the SCAN stage
  --stage SCAN \
  # Where the document are located
  --dataDir /path/to/documents \
  # Store the queued files in Redis
  --dataBusType REDIS \
  # URI of Redis 
  --redisAddress redis://redis:6379

2. INDEX

datashare --mode CLI \
  # Select the INDEX stage
  --stage INDEX \
  # Where the document are located
  --dataDir /path/to/documents \
  # Store the queued files in Redis
  --dataBusType REDIS \
  # URI of Elasticsearch
  --elasticsearchAddress http://elasticsearch:9200 \
  # Enable OCR \
  --ocr true
  # URI of Redis 
  --redisAddress redis://redis:6379

3. NLP

Once a document is available for search (stored in ElasticSearch), you can use the NLP stage to extract named entities from the text. This process will not only create named entities mentions in ElasticSearch, it will mark every analyzed document with the corresponding NLP pipeline (CORENLP by default). In other words, the process is idempotent and can be paralelized as well on several servers.

datashare --mode CLI \
  # Select the NLP stage
  --stage NLP \
  # Use CORENLP to detect named entities
  --nlpp CORENLP \
  # URI of Elasticsearch
  --elasticsearchAddress http://elasticsearch:9200 

Web modes

Those two modes are the only one who create

In local mode and embedded mode, Datashare provides a self-contained software application that users can install and run on their own local machines. The software allows users to search into their documents within their own local environments, without relying on external servers or cloud infrastructure. This mode offers enhanced data privacy and control, as the datasets and analysis remain entirely within the user's own infrastructure.

In server mode, Datashare operates as a centralized server-based system. Users can access to the platform through a web interface, and the documents are stored and processed on Datashare's servers. This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.

Comparaison between modes

The running modes offer advantages and limitations. This matrix summarizes the differences:

When running Datashare in local mode, users can choose to use embedded services (like ElasticSearch, SQLITE, in-memory key/value store) on the same JVM than Datashare. This variant of the local mode is called "embedded mode" and allows user to run Datashare without having to setup any additional software. The embedded mode is used by default.

CLI mode

In cli mode, Datashare starts without a web server and allow user to perform task over their documents. This mode can be used in conjunction both with local and server modes, while allowing users to distribute heaving task between several servers.

Daemon modes

Those modes are intended to be used for action that requires to "wait" for pendings tasks.

In batch download mode, the daemon wait for a user to request a batch download of documents. When a request is receive, the daemon start a task to download the document matching the user search, a bundle them into a zip file.

In batch search mode, the daemon wait for a user to request a batch search of documents. To create a batch search, users must go through the dedicated form on Datashare where they can to upload a list of search terms (in CSV format). The daemon will then start a task to search all matching document and store every occurences in the database.

How to change modes

Datashare is shipped as a single executable, with all modes available. As previously mentioned, the default mpode is the embedded mode. Yet when starting Datashare in command line, you can explicitely specify the running mode. For instance on Ubuntu/Debian:

datashare \
  # Switch to SERVER mode
  --mode SERVER \
  # Dummy session filter to creates ephemeral users
  --authFilter org.icij.datashare.session.YesCookieAuthFilter \
  # Name of the default project for every user
  --defaultProject local-datashare \
  # URI of Elasticsearch
  --elasticsearchAddress http://elasticsearch:9200 \
  # URI of Redis 
  --redisAddress redis://redis:6379 \
  # store user sessions in Redis.
  --sessionStoreType REDIS

Once Datashare is installed, go to "Finder", then "Applications", and double-click on "Datashare".

A Terminal window called 'Datashare.command' opens and describes the technical operations going on during the opening.

Keep this Terminal window open as long as you use Datashare.

When you open your desktop, you will see a folder called 'Datashare Data'. Move or copy and paste the documents you want to add to Datashare to this folder:

Once Datashare has opened, click on 'Analyze documents' on the top navigation bar in Datashare:​

Currently, only a .deb package for Debian/Ubuntu is provided.

1. Download Datashare

Save the Debian package as a file

2. Install the package

3. Run Datashare with:

The installer will setup:

• MacPorts (if neither Homebrew or MacPorts are installed)

• Apache Tesseract with MacPorts or Homenrew

• Java JRE 17

• Datashare executable

1. Download Datashare

2. Start the installer

Go to your "Downloads" directory in Finder and double-click "datashare-X.Y.Z.pkg":

3. Go through the Datashare Installer

Click 'Continue', 'Install', enter your password and 'Install Software':

The installation begins. You see a progress bar. It stays a long time on "Running package scripts" because it is installing XCode Command Line Tools, MacPort, Tesseract OCR, Java Runtime Environment and finally Datashare.

You can see what it actually does by typing command+L, it will open a window which logs every action made.

In the end, you should see this screen:

Standard (Windows 7 Service Pack 2 or newer version)

Uninstall any prior standard version

Download Datashare

The file "datashare-X.Y.Z.exe" is now downloaded. Double click on the name of the file in order to execute it.

Allow Datashare to install

As Datashare is not signed, this popup asks for your permission. Don't click 'Don't run' but click 'More info':

Click 'Run anyway':

It asks if you want to allow the app to make changes to your device. Click 'Yes':

Install Datashare

On the Installer Wizard, as you need to download and install OpenJDK11 if it is not installed on your device, click 'Install':

The following windows with progress bars will be displayed:

Choose a language and click 'OK':

Install Tesseract OCR

To install Tesseract OCR, click the following buttons on the Installer Wizard's windows:

Untick 'Show README' and click 'Finish':

Finally, click "Close" to close the installer of TesseractOCR.

Install Datashare.jar

It now downloads the back end and the front end, Datashare.jar:

When it is finished, click 'Close':

To be able to perform OCR, Datashare uses an open source technology called Apache Tesseract. When Tesseract extracts text from images, it uses "language packages" especially trained for each specific languages. Unfortunately, those packages can be heavy and to ensure a lightweight installation of Datashare, the installer doesn't use them all by default. In the case Datashare informs you of a missing package, this guide will explain you how to manually install it on your system.

Install packages on Linux

To add ocr languages on linux, simply use the following command:

sudo apt install tesseract-ocr-[lang]

Where `[lang]` is can be :

• all if you want to install all languages

Install packages on Mac

With MacPorts (default)

First, you must check that MacPort is installed on your computer. Please run in a Terminal:

port version

You should see an output similar to this:

If MacPort is installed on your computer, you should be able to add the missing Tesseract language package with the following command (for German):

port install tesseract-deu

Once the installation is done, simply close and restart Datashare to be able to use the newly installed packages.

With Homebrew

If Homebrew was already present on your system when Datashare was installed, Datashare used it to install Tesseract and its language packages. Because Homebrew doesn't package each Tesseract language individually, all languages are already supported by your system. In other words, you have nothing to do!

If you want to check if Homebrew is installed, run the following command in a Terminal:

brew -v

You should see an output similar to this:

Install languages on Windows

*Additionnal languages can be also added during Tesseract's installation.

The list of installed languages can be checked with Windows command prompt or Powershell with the commandtesseract --list-langs.

Datashare has to be restarted after the language installation.

Open the Windows main menu at the left of the bar at the bottom of your computer screen and click on 'Datashare'. (The numbers after 'Datashare' just indicate which version of Datashare you installed.)

A window called 'Terminal' will have opened, showing the progress of opening Datashare. Do not close this black window as long as you use Datashare.

Keep this Terminal window open as long as you use Datashare.

Start Datashare by launching it from the command-line:

1. You can find a folder called 'Datashare' in your home directory.

2. Move the documents you want to add to Datashare into this folder.

3. Open Datashare to extract text and eventually find people, organizations and locations in your documents.

Prerequisites

Get neo4j up and running

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the 'Other platforms and versions' button when downloading to access versions if necessary.

Add entities

If your project contains email documents, make sure to also extract email addresses.

Next step

Add documents

1. To add your documents in Datashare, click 'Tasks' in the left menu:

2. Click 'Analyze your documents':

3. Click 'Add documents' so Datashare can extract the texts from your files:

Options when adding documents

You can:

• Select the specific folder or sub-folder containing the documents you want to add.

• Extract text also from images/PDFs (OCR). Be aware the indexing can be up to 10 times longer.

• Skip already indexed files.

Two extraction tasks are now running: the first is the scanning of your Datashare folder which sees if there are new documents to analyze (ScanTask). The second is the indexing of these files (IndexTask):

It is not possible to 'Find people, organizations and locations' while of these two tasks is still running.

When tasks are done, you can start exploring documents by clicking 'Search' in the left menu but you won't have the named entities (names of people, organizations and locations) yet. To have these, follow the steps below.

Extract names of people, organizations and locations

1. After the text is extracted, you can launch named entities recognition by clicking the button 'Find people, organizations and locations'.

2. In the window below, you are asked to choose between finding Named Entities or finding email addresses (you cannot do both simultaneously, you need to do one after the other, no matter the order):

You can now see running tasks and their progress. After they are done, you can click 'Clear done tasks' to stop displaying tasks that are completed.

3. You can search your indexed documents without having to wait for all tasks to be done. To access your documents, click 'Search':

Extract email addresses

To extract email addresses in your documents:

• Re-click on 'Find people, organizations, locations and email addresses' (in Tasks (left menu) > Analyze your documents)

• Click the second radio button 'Find email addresses':

Authentication with Datashare in server mode is the most impacting choice that has to be made. It can be one of the followings:

• basic authentication with credentials stored in database (PostgreSQL)

• basic authentication with credentials stored in Redis

• OAuth2 with credentials provided by an identity provider (KeyCloak for example)

• dummy basic auth to accept any user (⚠️ if the service is exposed to internet, it will leak your documents)

Add plugins to Datashare (front-end)

1. Go to "Settings":

1. Click "Plugins":

1. Choose the plugin you want to add and click "Install now":

If you want to install a plugin from an URL, click "Install plugin from URL".

1. Your plugin is installed.

Refresh your page to see your new plugin activated in Datashare.

Add extensions to Datashare (back-end)

1. Go to "Settings":

1. Click "Extensions":

1. Choose the extension you want to add and click "Install now":

If you want to install an extension from an URL, click "Install extension from URL".

1. Your extension is installed.

Restart Datashare to see your new extension activated in Datashare.

Update plugin or extension with latest version

When a newer version of a plugin or extension is available, you can click on the "Update" button to get the latest version.

After that, if it is a plugin, refresh your page to activate the latest version.

If it is an extension, restart Datashare to activate the latest version.

Create your own plugin or extension

People who code can create their own plugins and extensions by following these steps:

This is likelly to be changed in the near future, but in the meantime, you can still add documents to Datashare using the command-line interface.

Here is a simple command to scan a directory and index its files:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage SCAN,INDEX \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --dataDir /home/datashare/Datashare/

What's happening here:

• The SCAN stage feeds a queue in memory with file to add

• The INDEX stage pulls files from the queue to add them to ElasticSearch

• We tell Datashare to use the elasticsearch service

• Files to add are located in /home/datashare/Datashare/ which is a directory mounted from the host machine

Alternativly, you can do this in two separated phases, as long as you tell Datashare to store the queue in a shared resource. Here, we use the redis:

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage SCAN \
  --queueType REDIS \
  --queueName "datashare:queue" \
  --redisAddress redis://redis:6379 \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --dataDir /home/datashare/Datashare/

Once the opperation is done, we can easily check the content of queue created by Datashare in redis. In this example we only display the 20 first files in the datashare:queue:

docker compose exec redis redis-cli lrange datashare:queue 0 20

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage INDEX \
  --queueType REDIS \
  --queueName "datashare:queue" \
  --redisAddress redis://redis:6379 \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --dataDir /home/datashare/Datashare/

Once the indexing is done, Datashare will exit gracefully and your document will already be visible on Datashare.

Sometimes you will face the case where you have an existing index, and you want to index additional documents inside your working directory without processing every document again. It can be done in two steps :

• Scan the existing ElasticSearch index and gather document paths to store it inside a report queue

• Scan and index (with OCR) the documents in the directory, thanks to the previous report queue, it will skip the paths inside of it

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage SCANIDX \
  --queueType REDIS \
  --reportName "report:queue" \
  --redisAddress redis://redis:6379 \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --dataDir /home/datashare/Datashare/

docker compose exec datashare_web /entrypoint.sh \
  --mode CLI \
  --stage SCAN,INDEX \
  --ocr true \
  --queueType REDIS \
  --queueName "datashare:queue" \
  --reportName "report:queue" \
  --redisAddress redis://redis:6379 \
  --defaultProject secret-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --dataDir /home/datashare/Datashare/

Install the neo4j plugin

Configure the neo4j plugin

1. Go to "Settings":

2. Make sure the following settings are properly set:

• Neo4j Host should be localhost or the address where your neo4j instance is running

• Neo4j Port should be the port where your neo4j instance is running (7687 by default)

• Neo4j User should be set to your neo4j user name (neo4j by default)

• Neo4j Password should only be set if your neo4j user is using password authentication

3. When running Neo4j Community Edition, set the Neo4j Single Project value. In community edition, the neo4j DBMS is restricted to a single database. Since Datashare supports multiple projects, you must set the Neo4j Single Project with the name of the project which will use neo4j plugin. Other projects won't be able to use the neo4j plugin.

4. Restart Datashare to apply the changes

5. You should be able to see the neo4j widget in your project page, after a little while its status should be RUNNING:

Next step

Prerequisites

Datashare platform is designed to function effectively by utilizing a combination of various services. To streamline the development and deployment workflows, Datashare relies on the use of Docker containers. Docker provides a lightweight and efficient way to package and distribute software applications, making it easier to manage dependencies and ensure consistency across different environments.

Starting Datashare with a single container

docker run --mount src=$HOME/Datashare,target=/home/datashare/data,type=bind -p 8080:8080 icij/datashare:11.1.9 --mode EMBEDDED

Make sure the Datashare folder exists in your homedir or this command will fail. This is an example about how to use Datashare with Docker, data will not be persisted.

Starting Datashare with multiple containers

Within Datashare, Docker Compose can play a significant role in enabling the setup of separated and persistent services for essential components such as the database (PostgreSQL), the search index (Elasticsearch), and the key-value store (Redis).

By utilizing Docker Compose, you can define and manage multiple containers as part of a unified service. This allows for seamless orchestration and deployment of interconnected services, each serving a specific purpose within the Datashare ecosystem.

Specifically, Docker Compose allows you to configure and launch separate containers for PostgreSQL, Elasticsearch, and Redis. These containers can be set up in a way that ensures their data is persistently stored, meaning that any information or changes made to the database, search index, or key-value store will be retained even if the containers are restarted or redeployed.

This separation of services using Docker Compose provides several advantages. It enhances modularity, scalability, and maintainability within the Datashare platform. It allows for independent management and scaling of each service, facilitating efficient resource utilization and enabling seamless upgrades or replacements of individual components as needed.

version: "3.7"
services:

  datashare:
    image: icij/datashare:18.1.3
    hostname: datashare
    ports:
      - 8080:8080
    environment:
      - DS_DOCKER_MOUNTED_DATA_DIR=/home/datashare/data
    volumes:
      - type: bind
        source: ${HOME}/Datashare
        target: /home/datashare/data
      - type: volume
        source: datashare-models
        target: /home/datashare/dist
    command: >-
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password 
      --mode LOCAL
      --tcpListenPort 8080
    depends_on:
      - postgresql
      - redis
      - elasticsearch

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
    restart: on-failure
    volumes:
      - type: volume
        source: elasticsearch-data
        target: /usr/share/elasticsearch/data
        read_only: false
    environment:
      - "http.host=0.0.0.0"
      - "transport.host=0.0.0.0"
      - "cluster.name=datashare"
      - "discovery.type=single-node"
      - "discovery.zen.minimum_master_nodes=1"
      - "xpack.license.self_generated.type=basic"
      - "http.cors.enabled=true"
      - "http.cors.allow-origin=*"
      - "http.cors.allow-methods=OPTIONS, HEAD, GET, POST, PUT, DELETE"

  redis:
    image: redis:4.0.1-alpine
    restart: on-failure

  postgresql:
    image: postgres:12-alpine
    environment:
      - POSTGRES_USER=datashare
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=datashare
    volumes:
      - type: volume
        source: postgresql-data
        target: /var/lib/postgresql/data

volumes:
  datashare-models:
  elasticsearch-data:
  postgresql-data:

Open a terminal or command prompt and navigate to the directory where you saved the docker-compose.yml file. Then run the following command to start the Datashare service:

docker-compose up -d

The -d flag runs the containers in detached mode, allowing them to run in the background.

Docker Compose will pull the necessary Docker images (if not already present) and start the containers defined in the YAML file. Datashare will take a few seconds to start. You can check the progression of this opperation with:

docker-compose logs -f datashare

Once the containers are up and running, you can access the Datashare service by opening a web browser and entering http://localhost:8080. This assumes that the default port mapping of 8080:8080 is used for the Datashare container in the YAML file.

That's it! You should now have the Datashare service up and running, accessible through your web browser. Remember that the containers will continue to run until you explicitly stop them.

To stop the Datashare service and remove the containers, you can run the following command in the same directory where the docker-compose.yml file is located:

docker-compose down

This will stop and remove the containers, freeing up system resources.

In server mode, Datashare operates as a centralized server-based system. Users can access to the platform through a web interface, and the documents are stored and processed on Datashare's servers. This mode offers the advantage of easy accessibility from anywhere with an internet connection, as users can log in to the platform remotely. It also facilitate seamless collaboration among users, as all the documents and analysis are centralized.

Launch configuration

Datashare is launched with --mode SERVER and you have to provide:

• the external elasticsearch index address elasticsearchAddress

• a Redis store address redisAddress

• a Redis data bus address messageBusAddress

• the host of Datashare (used to generate batch search results URLs) rootHost

• an authentication mechanism and its parameters

Example:

docker run -ti ICIJ/datashare:version --mode SERVER \
    --redisAddress redis://my.redis-server.org:6379 \
    --elasticsearchAddress https://my.elastic-server.org:9200 \
    --messageBusAddress my.redis-server.org \
    --dataSourceUrl jdbc:postgresql://db-server/ds-database?user=ds-user&password=ds-password \
    --rootHost https://my.datashare-server.org
    # ... +auth parameters (see authentication providers section)

Find your Datashare folder on your Mac

Open your Mac's 'Finder' by clicking on the blue smiling icon in your Mac's 'Dock':

On the menu bar at the top of your computer, click 'Go'. Click on 'Home' (the house icon).

You will see a folder called 'Datashare':

If you want to quickly access it in the future, you can drag and drop it in 'Favorites' on the left of this window:

Add documents in your Datashare folder

Copy or place the documents you want to have in Datashare in this Datashare folder.

Launch Datashare

Open your Applications. You should see Datashare. Double click on it:

Datashare opens in your default internet browser. Click 'Tasks':

Click the 3rd tab 'Analyze your documents':

Basic authentication is a simple protocol that uses the HTTP headers and the browser to authenticate users. User credentials are sent to the server in the header Authorization with user:password base64 encoded:

It is secure as long as the communication to the server is encrypted (with SSL for example).

On the server side, you have to provide a database user inventory. You can launch datashare first with the full database URL, then datashare will automatically migrate your database schema. Datashare supports SQLite and PostgreSQL as back-end databases. SQLite is not recommended for a multi-user server because it cannot be multithreaded, so it will introduce contention on users' DB SQL requests.

Then you have to provision users. The passwords are sha256 hex encoded (for example with bash):

Then you can insert the user like this in your database:

If you use other indices, you'll have to include them in the group_by_applications, but local-datashare should remain. For exammple if you use myindex:

Then when accessing Datashare, you should see this popup:

Example

Here is an example of launching Datashare with Docker and the basic auth provider filter backed in database:

Prerequisites

Datashare platform is designed to function effectively by utilizing a combination of various services. To streamline the development and deployment workflows, Datashare relies on the use of Docker containers. Docker provides a lightweight and efficient way to package and distribute software applications, making it easier to manage dependencies and ensure consistency across different environments.

Starting Datashare with multiple containers

Within Datashare, Docker Compose can play a significant role in enabling the setup of separated and persistent services for essential components. By utilizing Docker Compose, you can define and manage multiple containers as part of a unified service. This allows for seamless orchestration and deployment of interconnected services, each serving a specific purpose within the Datashare ecosystem.

Specifically, Docker Compose allows you to configure and launch separate containers for PostgreSQL, Elasticsearch, and Redis. These containers can be set up in a way that ensures their data is persistently stored, meaning that any information or changes made to the database, search index, or key-value store will be retained even if the containers are restarted or redeployed.

This separation of services using Docker Compose provides several advantages. It enhances modularity, scalability, and maintainability within the Datashare platform. It allows for independent management and scaling of each service, facilitating efficient resource utilization and enabling seamless upgrades or replacements of individual components as needed.

Open a terminal or command prompt and navigate to the directory where you saved the docker-compose.yml file. Then run the following command to start the Datashare service:

The -d flag runs the containers in detached mode, allowing them to run in the background.

Docker Compose will pull the necessary Docker images (if not already present) and start the containers defined in the YAML file. Datashare will take a few seconds to start. You can check the progression of this opperation with:

Once the containers are up and running, you can access the Datashare service by opening a web browser and entering http://localhost:8080. This assumes that the default port mapping of 8080:8080 is used for the Datashare container in the YAML file.

To stop the Datashare service and remove the containers, you can run the following command in the same directory where the docker-compose.yml file is located:

This will stop and remove the containers, freeing up system resources.

Add documents to Datashare

Extract named entities

This is likely to be changed in the near future, but in the meantime, you can extract named entities using the command-line interface.

What's happening here:

• We tell Datashare to use the elasticsearch service

• Datashare will pull documents from ElasticSearch directly

• Up to 2 documents will be analyzed in parallel

• Datashare will use the CORENLP pipeline

Datashare will use the output queue from the previous INDEX stage (by default extract:queue:nlp in Redis) that contains all the document ids to be analyzed.

The first time you run this command you will have to wait a little bit because Datashare need to download CORENLP's models which can be big.

You can also use chain the 3 stages altogether:

The added ENQUEUEIDX stage will read Elasticsearch index, find all documents that have not already been analyzed by the CORENLP NER pipeline, and put the ids of those documents into the extract:queue:nlp queue.

Create the graph

1. Open the 'Projects' page and select your project:

1. Create the graph by clicking on the 'Create graph' button inside the neo4j widget:

You will see a new import task running:

When the graph creation is complete, 'Graph statistics' will reflect the number of documents and entities nodes found in the graph:

Update the graph

When new documents or entities are added or modified inside Datashare, you will need to update the neo4j graph to reflect these changes.

To update the graph click on the 'Update graph' button inside the neo4j widget:

To detect whether a graph update is needed you can compare the number of documents found inside Datashare to the number found in the 'Graph statistics' and run an update in case of mismatch:

The update will always add missing nodes and relationships, update existing ones if they were modified, but will never delete graph nodes or relationships.

Next steps

This is the default authentication mode: if not provided in CLI it will be selected. With OAuth2 you will need a third-party authorization service. The diagram below describes the workflow:

Example

Integration with KeyCloak

Run the neo4j extension CLI

The neo4j related features are added to the DatashareCLI through the extension mechanism. In order to run the extended CLI, the Java CLASSPATH must be extended with the path of the datashare-extension-neo4j jar. By default, this jar is located in /home/datashare/extensions, so the CLI will be run as following:

docker compose exec \
  -e CLASSPATH=/home/datashare/extensions/* \
  datashare_web /entrypoint.sh \
  --mode CLI \
  --ext neo4j \
  ... 

Create the graph

In order to create the graph, run the --fullImport command for your project:

docker compose exec \
  -e CLASSPATH=/home/datashare/extensions/* \
  datashare_web /entrypoint.sh \
  --mode CLI \
  --ext neo4j \
  --full-import \
  --project secret-project

the CLI will display the import task progress and log import related information.

Update the graph

When new documents or entities are added or modified inside Datashare, you will need to update the neo4j graph to reflect these changes.

To update the graph, you can just re-run the full export:

docker compose exec \
  -e CLASSPATH=/home/datashare/extensions/datashare-extension-*.jar \
  datashare_web /entrypoint.sh \
  --mode CLI \
  --ext neo4j \
  --full-import \
  --project secret-project

The update will always add missing nodes and relationships, update existing ones if they were modified, but will never delete graph nodes or relationships.

To detect whether a graph update is needed, open the 'Projects' page and select your project:

compare the number of documents and entities found inside Datashare:

to the numbers found in the 'Graph statistics' and run an update in case of mismatch:

Next steps

Search in documents

If not collapsed yet, to collapse the left menu in order to gain room, click the 'hamburger menu':

2. Search for specific documents. Type terms in the search bar, press Enter or click 'Search':

IMPORTANT:

3. You can search in specific fields like tags, title, author, recipient, content, path or thread ID. Click 'All fields' and select your choice in the dropdown menu:

Choose between views (list, grid, table)

Select the view on the top right.

• List:

• Grid:

• Table:

Search within a document

Once a document is opened, you can search for terms in this document:

• Press Command (⌘) + F (on Mac) or Control + F (on Windows and Linux) or click on the search bar above your Extracted Text

• Type what you search for

• Press ENTER to go from one occurrence to the next one

• Presse SHIFT + ENTER to go from one occurrence to the previous one

This also counts the number of occurrences of your searched terms in this document:

Generated with https://github.com/ICIJ/fluent-http-apigen

Table of Content

/api/key

Operation /api/keydescription

ApiResponse /api/keyresponseCode

Options /api/key/:userId

Operation /api/keydescription

ApiResponse /api/keyresponseCode

Put /api/key/:userId

Operation /api/keydescription

ApiResponse /api/keyresponseCode

Get /api/key/:userId

Operation /api/keydescription

ApiResponse /api/keyresponseCode

Delete /api/key/:userId

/api/batch

Get /api/batch/search

Retrieve the batch search list for the user issuing the request.

• Return 200 and the list of batch searches

Example :

Post /api/batch/search

Retrieve the batch search list for the user issuing the request filter with the given criteria, and the total of batch searches matching the criteria.

It needs a Query json body with the parameters :

• from : index offset of the first document to return (mandatory)

• size : window size of the results (mandatory)

• sort : field to sort (prj_id name user_id description state batch_date batch_results published) (default "batch_date")

• order : "asc" or "desc" (default "asc")

• project : projects to include in the filter (default null / empty list)

• batchDate : batch search with a creation date included in this range (default null / empty list)

• state : states to include in the filter (default null / empty list)

• publishState : publish state to filter (default null)

If from/size are not given their default values are 0, meaning that all the results are returned. BatchDate must be a list of 2 items (the first one for the starting date and the second one for the ending date) If defined publishState is a string equals to "0" or "1"

• Return 200 and the list of batch searches with the total batch searches for the query. See example for the JSON format.

Example :

Get /api/batch/search/:batchid

Retrieve the batch search with the given id The query param "withQueries" accepts a boolean value When "withQueries" is set to false, the list of queries is empty and nbQueries contains the number of queries.

• Parameter batchId

• Return 200 and the batch search

Example :

Get /api/batch/search/:batchid/queries

Retrieve the batch search queries with the given batch id and returns a list of strings UTF-8 encoded

if the request parameter format is set with csv, then it will answer with content-disposition attachment (file downloading)

the optional request parameters are :

• from: if not provided it starts from 0

• size: if not provided all queries are returned from the "from" parameter

• search: if provided it will filter the queries accordingly

• orderBy: field name to order by asc, "query_number" by default (if it does not exist it will return a 500 error)

• maxResult: number of maximum results for each returned query (-1 means no maxResults)

• Parameter batchId

• Return 200 and the batch search queries map [(query, nbResults), ...]

Example :

Options /api/batch/search

preflight request

• Return 200 DELETE

Options /api/batch/search/:batchid

preflight resquest for removal of one batchsearch

• Parameter batchId

• Return 200 DELETE

Delete /api/batch/search/:batchid

Delete batch search with the given id and its results. It won't delete running batch searches, because results are added and would be orphans.

Returns 204 (No Content) : idempotent

• Return 204

Example :

Patch /api/batch/search/:batchid

Update batch search with the given id.

Returns 200 and 404 if there is no batch id If the user issuing the request is not the same as the batch owner in database, it will do nothing (thus returning 404)

• Return 200 or 404

Example :

Post /api/batch/search/:coma

Creates a new batch search. This is a multipart form with 8 fields : name, description, csvFile, published, fileTypes, paths, fuzziness, phrase_matches

No matter the order. The name and csv file are mandatory else it will return 400 (bad request) Csv file must have under 60 000 lines else it will return 413 (payload too large) Queries with less than two characters are filtered

To do so with bash you can create a text file like :

Then replace with with a sed like this:

sed -i 's/$/^M/g' ~/multipart.txt

Then make a curl request with this file :

• Parameter comaSeparatedProjects

• Parameter context : the request body

• Return 200 or 400 or 413

Options /api/batch/search/copy/:sourcebatchid

preflight request

• Return 200 POST

Post /api/batch/search/copy/:sourcebatchid

Create a new batch search based on a previous one given its id, and enqueue it for running

it returns 404 if the source BatchSearch object is not found in the repository.

• Parameter sourceBatchId: the id of BatchSearch to copy

• Parameter context : the context of request (containing body)

• Return 200 or 404

Example:

Post /api/batch/search/result/:batchid

Retrieve the results of a batch search as JSON.

It needs a Query json body with the parameters :

• from : index offset of the first document to return (mandatory)

• size : window size of the results (mandatory)

• queries: list of queries to be downloaded (default null)

• sort: field to sort ("doc_nb", "doc_id", "root_id", "doc_path", "creation_date", "content_type", "content_length", "creation_date") (default "doc_nb")

• order: "asc" or "desc" (default "asc")

If from/size are not given their default values are 0, meaning that all the results are returned.

• Parameter batchId

• Parameter webQuery

• Return 200

Example :

Get /api/batch/search/result/csv/:batchid

Retrieve the results of a batch search as a CSV file.

The search request is by default all results of the batch search.

• Parameter batchId

• Return 200 and the CSV file as attached file

Example :

Delete /api/batch/search

Delete batch searches and results for the current user.

Returns 204 (No Content): idempotent

• Return 204

Example :

/api

Get /api/:project/documents/src/:id?routing=:routing

Returns the file from the index with the index id and the root document (if embedded document).

The routing can be omitted if it is a top level document, or it can be the same as the id.

Returns 404 if it doesn't exist

Returns 403 if the user has no access to the requested index.

• Parameter project

• Parameter id

• Parameter routing

• Return 200 or 404 or 403 (Forbidden)

Example :

Get /api/:project/documents/content/:id?routing=:routing

Fetch extracted text by slice (pagination)

• Parameter project Project id

• Parameter id Document id

• Parameter offset Starting byte (starts at 0)

• Parameter limit Size of the extracted text slice in bytes

• Parameter targetLanguage Target language (like "ENGLISH") to get slice from translated content

• Return 200 and a JSON containing the extracted text content ("content":text), the max offset as last rank index ("maxOffset":number), start ("start":number) and size ("size":number) parameters.

• Throws IOException

Example :

Get /api/:project/documents/searchContent/:id?routing=:routing

Search query occurrences in content or translated content (pagination)

• Parameter project Project id

• Parameter id Document id

• Parameter query Query string to search occurrences (starts at 0)

• Parameter targetLanguage Target language (like "ENGLISH") to search in translated content

• Return 200 and a JSON containing the occurrences offsets in the text, and the count of occurrences.

• Throws IOException

Example :

Post /api/:project/documents/batchUpdate/star

Group star the documents. The id list is passed in the request body as a json list.

It answers 200 if the change has been done and the number of documents updated in the response body.

• Parameter projectId

• Parameter docIds as json

• Return 200 and the number of documents updated

Example :

Post /api/:project/documents/batchUpdate/unstar

Group unstar the documents. The id list is passed in the request body as a json list.

It answers 200 if the change has been done and the number of documents updated in the response body.

• Parameter projectId

• Parameter docIds as json in body

• Return 200 and the number of documents unstarred

Example :

Get /api/:project/documents/starred

Retrieves the list of starred document for a given project.

• Parameter projectId

• Return 200

Example :

Get /api/:projects/documents/tagged/:coma

Retrieves the list of tagged document with tag "tag" for the given project id.

This service doesn't need to have the document stored in the database (no join is made)

• Parameter projectId

• Parameter comaSeparatedTags

• Return 200

Example :

Options /api/:project/documents/tags/:docId

preflight request

• Parameter projectId

• Parameter docId

• Return 200 PUT

Put /api/:project/documents/tags/:docId?routing=:routing

• Parameter projectId

• Parameter docId

• Parameter routing

• Parameter tags

• Return 201 if created else 200

Example :

Get /api/:project/documents/tags/:docId

Gets all the tags from a document with the user and timestamp.

• Parameter projectId

• Parameter docId

• Return 200 and the list of tags

Example :

Post /api/:project/documents/batchUpdate/tag

Group tag the documents. The document id list and the tag list are passed in the request body.

It answers 200 if the change has been done.

• Parameter projectId

• Parameter query

• Return 200

Example :

Post /api/:project/documents/batchUpdate/untag

Group untag the documents. The document id list and the tag list are passed in the request body.

It answers 200 if the change has been done.

• Parameter projectId

• Parameter query

• Return 200

Example :

Options /api/:project/documents/untag/:docId

preflight request

• Parameter projectId

• Parameter docId

• Return 200 PUT

Put /api/:project/documents/untag/:docId?routing=:routing

Untag one document

• Parameter projectId

• Parameter docId

• Parameter routing

• Parameter tags

• Return 201 if untagged else 200

Get /api/documents/starred

Retrieves the list of starred document for all projects.

This service needs to have the document stored in the database.

• Return 200 and the list of Documents

Get /api/users/recommendations?project=:project

Retrieves the list of users who recommended a document with the total count of recommended documents for the given project id

• Parameter projectId

• Return 200

Example :

Get /api/users/recommendationsby?project=:project

Get all users who recommended a document with the count of all recommended documents for project and documents ids.

• Parameter projectId

• Parameter comaSeparatedDocIds

• Return 200 and the list of tags

Example :

Get /api/:project/documents/recommendations?userids=:coma

Retrieves the set of marked read documents for the given project id and a list of users provided in the url.

This service doesn't need to have the document stored in the database (no join is made)

• Parameter projectId

• Parameter comaSeparatedUsers

• Return 200

Example :

Post /api/:project/documents/batchUpdate/recommend

Group mark the documents "read". The id list is passed in the request body as a json list.

It answers 200 if the change has been done and the number of documents updated in the response body.

• Parameter projectId

• Parameter docIds as json

• Return 200 and the number of documents marked

Example :

Post /api/:project/documents/batchUpdate/unrecommend

Group unmark the documents. The id list is passed in the request body as a json list.

It answers 200 if the change has been done and the number of documents updated in the response body.

• Parameter projectId

• Parameter docIds as json

• Return 200 and the number of documents unmarked

Example :

/api/extensions

Get /api/extensions

Gets the extension set in JSON

If a request parameter "filter" is provided, the regular expression will be applied to the list.

see https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html for pattern syntax.

Example:

• Return

Options /api/extensions/install

Preflight request

• Return OPTIONS,PUT

Put /api/extensions/install

Download (if necessary) and install extension specified by its id or url

request parameter id or url must be present.

• Return 200 if the extension is installed

• Return 404 if the extension is not found by the provided id or url

• Return 400 if neither id nor url is provided

• Throws IOException

Example:

Options /api/extensions/uninstall

Preflight request

• Return OPTIONS,DELETE

Delete /api/extensions/uninstall?id=:extensionId

Uninstall extension specified by its id

• Parameter extensionId

• Return 204 if the extension is uninstalled (idempotent)

• Throws IOException if there is a filesystem error

Example:

/api/index

Put /api/index/:index

Create the index for the current user if it doesn't exist.

• Return 201 (Created) or 200 if it already exists

Example :

Options /api/index/:index

Preflight for index creation.

• Parameter index

• Return 200 with PUT

Head /api/index/search/:path:

Head request useful for JS api (for example to test if an index exists)

• Parameter path

• Return 200

Post /api/index/search/:path:

The search endpoint is just a proxy in front of Elasticsearch, everything sent is forwarded to Elasticsearch. DELETE method is not allowed.

Path can be of the form :

• _search/scroll

• index_name/_search

• index_name1,index_name2/_search

• index_name/_count

• index_name1,index_name2/_count

• index_name/doc/_search

• index_name1,index_name2/doc/_search

• Parameter path

• Return 200 or http error from Elasticsearch

Example :

Get /api/index/search/:path:

Search GET request to Elasticsearch

As it is a GET method, all paths are accepted.

if a body is provided, the body will be sent to ES as source=urlencoded(body)&source_content_type=application%2Fjson in that case, request parameters are not taken into account.

• Parameter path

• Return 200 or http error from Elasticsearch

Example :

Options /api/index/search/:path:

Prefligth option request

• Parameter path

• Return 200

/api

Get /api/:project/namedEntities/:id?routing=:documentId

Returns the named entity with given id and document id.

• Parameter id

• Parameter documentId the root document

• Return 200

Example :

Options /api/:project/namedEntities/hide/:mentionNorm

preflight request for hide

• Parameter mentionNorm

• Return 200 PUT

Put /api/:project/namedEntities/hide/:mentionNorm

hide all named entities with the given normalized mention

• Parameter mentionNorm

• Parameter project

• Return 200

Example :

/api/ner

Get /api/ner/pipelines

Get the list of registered pipelines.

• Return pipeline set Example:

Post /api/ner/findNames/:pipeline

When datashare is launched in NER mode (without index) it exposes a name finding HTTP API. The text is sent with the HTTP body.

• Parameter pipeline to use

• Parameter text to analyse in the request body

• Return list of NamedEntities annotations

Example :

/api

Get /api/:project/notes/:path:

Gets the list of notes for a project and a document path.

if we have on disk:

And in database

then :

• GET /api/p1/notes/a/b/doc1 will return note A and B

• GET /api/p1/notes/a/c/doc2 will return note A

• GET /api/p1/notes/d/doc3 will return an empty list

If the user doesn't have access to the project she gets a 403 Forbidden

• Parameter project the project the note belongs to

• Parameter documentPath the document path

• Parameter context HTTP context containing the user

• Return list of Note that match the document path

Example:

Get /api/:project/notes

Gets the list of notes for a project.

If the user doesn't have access to the project she gets a 403 Forbidden

• Parameter project the project the note belongs to

• Parameter context HTTP context containing the user

• Return list of Note related to the project

Example:

/api/openapi

Operation /api/openapidescription

ApiResponse /api/openapiresponseCode

Get /api/openapi

/api/plugins

Get /api/plugins

Gets the plugins set in JSON

If a request parameter "filter" is provided, the regular expression will be applied to the list.

see https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html for pattern syntax.

Example:

Options /api/plugins/install

Preflight request

• Return OPTIONS,PUT

Put /api/plugins/install

Download (if necessary) and install plugin specified by its id or url

request parameter id or url must be present.

• Return 200 if the plugin is installed

• Return 404 if the plugin is not found by the provided id or url

• Return 400 if neither id nor url is provided

• Throws IOException

• Throws ArchiveException

Example:

Options /api/plugins/uninstall

Preflight request

• Return OPTIONS,DELETE

Delete /api/plugins/uninstall?id=:pluginId

Uninstall plugin specified by its id Always returns 204 or error 500.

• Parameter pluginId

• Return 204

• Throws IOException if there is a filesystem error

Example:

/api/project

Get /api/project/

Post /api/project/

Put /api/project/:id

Get /api/project/:id

Gets the project information for the given project id.

• Parameter id

• Return 200 and the project from database if it exists

Example :

``` curl -H 'Content-Type:application/json' localhost:8080/api/project/apigen-datashare {"error":"java.lang.NullPointerException"} ``` ) ##Get /api/project/isDownloadAllowed/:id Returns if the project is allowed with this network route : in datashare database there is the project table that can specify an IP mask that is allowed per project. If the client IP is not in the range, then the file download will be forbidden.

in that project table there is a field called allow_from_mask that can have a mask with IP and star wildcard.

Ex : 192.168.*.* will match all subnetwork 192.168.0.0 IP's and only users with an IP in this range will be granted for downloading documents.

• Parameter id

• Return 200 or 403 (Forbidden)

Example :

Example :

Options /api/project/:id

Preflight option request

• Parameter id

• Return 200 DELETE

Delete /api/project/:id

Delete the project from database and elasticsearch indices.

It always returns 204 (no content) or 500 if an error occurs.

If the project id is not the current user project (local-datashare in local mode), then it will return 401 (unauthorized)

• Parameter id

• Return 204

Example :

/

Get /

gets the root of the front-end app ie: ./app/index.html

if pluginsDir is set, it will add in the index the tag else it will return the index.html content as is

• Return the content of index.html file

Get /settings

Gets the public (i.e. without user's information) datashare settings parameters. These parameters are used for the client app for the init process.

The endpoint is removing all fields that contain Address or Secret or Url or Key

• Return 200

Example :

Get /version

Gets the versions (front/back/docker) of datashare.

• Return 200

Example :

/api/settings

Options /api/settings

Preflight for settings.

• Parameter context

• Return 200 with PATCH

Patch /api/settings

update the datashare settings with provided body. It will save the settings on disk.

Returns 404 if settings is not found. It means that the settings file has not been set (or is not readable) Returns 403 if we are in SERVER mode

The settings priority is basically DS_DOCKER_* variables > -s file > classpath:datashare.properties > command line. I.e. :

• DS_DOCKER_* variables will be taken and override all keys (if any similar keys exist)

• if a file is given (w/ -c path/to/file) to the command line it will be read and used (it can be empty or not present)

• if no file is given, we are looking for datashare.properties in the classpath (for example in /dist)

• if none of the two above cases is fulfilled we are taking the default CLI parameters (and those given by the user)

• parameters are common between CLI and settings file, the settings file "wins"

• if a settings file is not writable then 404 will be returned (and a WARN will be logged at start)

• Return 200 or 404 or 403

Example :

Get /api/settings/ocr/languages

List all available language in Tesseract

Returns 503 if Tesseract is not installed

• Return 200 or 503

Get /api/settings/text/languages

List all available language in the text extractor

• Return 200

/api

Get /api/status

Retrieve the status of databus connection, database connection, shared queues and index. Adding "format=openmetrics" parameter to the url will return the status witn openmetrics format.

• Return the status of datashare elements

Example:

/api/task

Get /api/task/all

gets all the user tasks a filter can be added with a pattern contained in the task name.

• Return 200 and the list of tasks

Example :

Get /api/task/:id

gets one task with its id

• Parameter id

• Return 200

Example :

Get /api/task/:id/result

gets task result with its id

• Parameter id

• Return 200 and the result, 204 if there is no result 404 if the tasks doesn't exist 403 if the task is not belonging to current user

Example :

Options /api/task/batchDownload

Post /api/task/batchDownload

download files from a search query. Expected parameters are :

• project: string

• query: string or elasticsearch JSON query

if the query is a string it is taken as an ES query string, else it is a raw JSON query (without the query part) @see org.elasticsearch.index.query.WrapperQueryBuilder that is used to wrap the query

• Parameter optionsWrapper wrapper for options json

• Return 200 and json task

Example :

Post /api/task/batchUpdate/index

index files from the queue

• Parameter optionsWrapper wrapper for options json

• Return 200 and json task

Example :

Post /api/task/batchUpdate/index/file

Indexes files in a directory (with docker, it is the mounted directory that is scanned)

• Parameter optionsWrapper

• Return 200 and the list of tasks created

Example :

Post /api/task/batchUpdate/index/:filePath:

Indexes all files of a directory with the given path.

• Parameter filePath

• Parameter optionsWrapper

• Return 200 and the list of created tasks

Example $(curl -XPOST localhost:8080/api/task/batchUpdate/index/home/dev/myfile.txt)

Post /api/task/batchUpdate/scan/:filePath:

Scans recursively a directory with the given path

• Parameter filePath

• Parameter optionsWrapper

• Return 200 and the task created

Example :

Post /api/task/clean

Cleans all DONE tasks.

• Return 200 and the list of removed tasks

Example :

Delete /api/task/clean/:taskName:

Cleans a specific task.

• Parameter taskName

• Return

Example :

Options /api/task/clean/:taskName:

Put /api/task/stop/:taskId:

Cancels the task with the given name. It answers 200 with the cancellation status true|false

• Parameter taskId

• Return

Options /api/task/stop/:taskName:

Put /api/task/stopAll

Cancels the running tasks. It returns a map with task name/stop statuses. If the status is false, it means that the thread has not been stopped.

• Return 200 and the tasks stop result map

Example : curl -XPUT localhost:8080/api/task/stopAll

Options /api/task/stopAll

Post /api/task/findNames/:pipeline

Find names using the given pipeline :

• OPENNLP

• CORENLP

• IXAPIPE

• GATENLP

• MITIE

This endpoint is going to find all Documents that are not taggued with the given pipeline, and extract named entities for all these documents.

• Parameter pipelineName

• Parameter optionsWrapper

• Return 200 and the list of created tasks

Example :

/api/tree

Get /api/tree:dirPath:

List all files and directory for the given path. This endpoint returns a JSON using the same specification than the tree command on UNIX. It is roughly the equivalent of:

• Parameter dirPath

• Return 200 and the list of files and directory

Example $(curl -XGET localhost:8080/api/tree/home/datashare/data)

/api/users

Get /api/users/me

Gets the user's session information

• Return 200 and the user map

Example :

Options /api/users/me/history

Preflight for history.

• Return 200 with OPTIONS, GET, PUT and DELETE

Get /api/users/me/history?type=:type

Gets the user's history by type

• Parameter type String included in 'document' or 'search'

• Parameter from the offset of the list, starting from 0

• Parameter size the number of element retrieved

• Parameter sort the name of the parameter to sort on (default: modificationDate)

• Parameter desc the list is sorted in descending order (default: true)

• Parameter projects projectIds separated by comma to filter by projects (default: none)

• Return 200, the user's list of events and the total number of events

Example : ``` curl -i localhost:8080/api/users/me/history?type=document&from=0&size=10&sort=modificationDate&desc=true&projects=project1,project2 HTTP/1.1 200 OK Access-Control-Allow-Origin: * Content-Type: application/json;charset=UTF-8 ETag: 9a3f093e2dc5d929bb25879501d527c7 Content-Length: 22 Connection: keep-alive Set-Cookie: _ds_session_id={"login":null,"roles":null,"sessionId":null,"redirectAfterLogin":"/"}; version=1; path=/; expires=Mon, 30-Jul-2091 14:00:32 GMT; max-age=2147483647

{"items":[],"total":0}

curl -i -XPUT -H "Content-Type: application/json" localhost:8080/api/users/me/history -d '{"type": "SEARCH", "projectIds": ["apigen-datashare","local-datashare"], "name": "foo AND bar", "uri": "?q=foo%20AND%20bar&from=0&size=100&sort=relevance&index=luxleaks&field=all&stamp=cotgpe"}' HTTP/1.1 500 Internal Server Error Content-Type: application/json;charset=UTF-8 ETag: b1b6023e69d8821fc4e1e8418ab85f30 Content-Length: 77 Connection: keep-alive

{"error":"org.jooq.exception.DataAccessException: Cannot commit transaction"}

curl -i -XDELETE localhost:8080/api/users/me/history?type=search HTTP/1.1 500 Internal Server Error Content-Type: application/json;charset=UTF-8 ETag: b1b6023e69d8821fc4e1e8418ab85f30 Content-Length: 77 Connection: keep-alive

{"error":"org.jooq.exception.DataAccessException: Cannot commit transaction"}

curl -i -XDELETE localhost:8080/api/users/me/history/event?id=1 HTTP/1.1 500 Internal Server Error Content-Type: application/json;charset=UTF-8 ETag: b1b6023e69d8821fc4e1e8418ab85f30 Content-Length: 77 Connection: keep-alive

{"error":"org.jooq.exception.DataAccessException: Cannot commit transaction"}

Basic authentication is a simple protocol that uses the HTTP headers and the browser to authenticate users. User credentials are sent to the server in the header Authorization with user:password base64 encoded:

It is secure as long as the communication to the server is encrypted (with SSL for example).

On the server side, you have to provide a user store for Datashare. For now we are using a Redis data store.

So you have to provision users. The passwords are sha256 hex encoded. For example using bash:

Then insert the user like this in Redis:

If you use other indices, you'll have to include them in the group_by_applications, but local-datashare should remain. For exammple if you use myindex:

Then you should see this popup:

Example

Here is an example of launching Datashare with Docker and the basic auth provider filter backed in Redis:

Prerequisites

Get neo4j up and running

We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the 'Other platforms and versions' button when downloading to access versions if necessary.

Add entities

If your project contains email documents, make sure to run the EMAIL pipeline together with regular NLP pipeline. To do so add set the follow nlpp flag to --nlpp CORENLP,EMAIL.

Next step

You can have a dummy authentication that always accepts basic auth. So you should see this popup:

But then you can enter whatever user or password, it will enter Datashare.

Example

Separate Processing Stages

Execute the SCAN and INDEX stages independently to optimize resource allocation and efficiency.

Examples:

Distribute the INDEX Stage

Leverage Parallelism

Datashare offer --parallelism and --parserParallelism options to enhance processing speed.

Example (for g4dn.8xlarge with 32 CPUs):

Optimize ElasticSearch