Download About ICIJ GitHub

Create a Neo4j graph and explore it

This page explains how to leverage neo4j to explore your Datashare projects. We recommend using a recent release of Datashare (>= 14.0.0) to use this feature, click on the "Other platforms and version

The documents and entities graph

neo4j is a graph database technology which lets you represent your data as a graph. Inside Datashare, neo4j lets you connect entities between them through documents in which they appear.

After creating a graph from your Datashare project, you will be able to explore it and visualize these kinds of relationships between you project entities:

In the above graph, we can see 3 email document nodes in orange, 3 email address nodes in red, 1 person node in green and 1 location node in yellow. Reading the relationship types on the arrows, we can deduce the following information from the graph:

  • shapp@caiso.com emailed 20participants@caiso.com, the sent email has an id starting with f4db344...

  • one person named vincent is mentioned inside this email, as well as the california location

  • finally, the email also mentions the dle@caiso.com email address which is also mentioned in 2 other email documents (with id starting with 11df197... and 033b4a2...)

If you are not familiar with graph and neo4j, take a look at the following resources:

Graph nodes

The neo4j graph is composed of :Document nodes representing Datashare documents and :NamedEntity nodes representing entities mentioned in these documents.

The :NamedEntity nodes are additionally annotated with their entity types: :NamedEntity:PERSON, :NamedEntity:ORGANIZATION, :NamedEntity:LOCATION, :NamedEntity:EMAIL...

Graph relationships

In most cases, an entity :APPEARS_IN a document, which means that it was detected in the document content. In the particular case of email documents and EMAIL addresses, it is most of the time possible to identify richer relationships from the email metadata, such as who sent (:SENT relationship) and who received (:RECEIVED relationship) the email.

When an :EMAIL address entity is neither :SENT or :RECEIVED, like it is the case in the above graph for dle@caiso.com, it means that the address was mentioned in the email document body.

When a document is embedded inside another document (as an email attachment for instance), the child document is connected to its parent through the :HAS_PARENT relationship.

Create your Datashare project's graph

The creation of a neo4j graph inside Datashare is supported through a plugin. To use the plugin to create a graph, follow these instructions:

After the graph is created, navigate to the 'Projects' page and select your project. You should be able to visualize a new neo4j widget displaying the number of documents and entities found inside the graph:

Access your project's graph

Depending on your access to the neo4j database behind Datashare, you might need to export the neo4j graph and import it locally to access it from visualization tools.

Exporting and importing the graph into your own DB is also useful when you want to perform write operations on your graph without any consequences on Datashare.

With read access to Datashare's neo4j database

If you have read access to the neo4j database (it should be the case if you are running Datashare on your computer), you will be able to plug visualization tools to it and start exploring.

Without read access to Datashare's neo4j database

If you can't have read access to the database, you will need to export it and import it into your own neo4j instance (running on your laptop for instance).

Ask for a DB dump

If it's possible, ask you system administrator for a DB dump obtained using the neo4j-admin database dump command.

Export your graph from Datashare

In case you don't have access to the DB and can't be provided with a dump, you can export the graph from inside. Be aware that limits might be applied on the size of the exported graph.

To export the graph, navigate to Datashare's 'Projects' page, select your project, select the 'Cypher shell' export format and click the 'Export graph' button:

In case you want to restrict the size of the exported graph, you can restrict the export to a subset of documents and their entities using the 'File types' and 'Project directory' filters.

DB import

Depending on how you run neo4j on your laptop use one of the following ways to import your graph into your DB:

Docker

  • identify your neo4j instance container ID:

    docker ps | grep neo4j # Should display your running neo4j container ID

  • copy your the graph dump inside your neo4j container import directory:

    docker cp \
    <export-path> \
    <neo4j-container-id>:/var/lib/neo4j/imports/datashare-graph.dump

  • import the dumped file using the cypher-shell command:

    docker exec -it <neo4j-container-id> /bin/bash
./bin/cypher-shell -f imports/datashare-graph.dump

Neo4j Desktop import

  • open 'Cypher shell':

  • copy your the graph dump inside your neo4j instance import directory:

    cp <export-path> imports

  • import the dumped file using the cypher-shell command:

    ./bin/cypher-shell -f imports/datashare-graph.dump

You will now be able to explore the graph imported in your own neo4j instance.

Explore and visualize entity links

Once your graph is created and that you can access it (see this section if you can't access the Datashare's neo4j instance), you will be able to use your favorite tool to extract meaningful information from it.

Connect to your database

Once you can access your neo4j database, you can use different tools to visualize and explore it. You can start by connection the Neo4j Desktop to your DB.

Visualize and explore with Neo4j Bloom

Neo4j Bloom is a simple and powerful tool developed by neo4j to quickly visualize and query graphs, if you run Neo4j Enterprise Edition. Bloom lets you navigate and explore the graph through a user interface similar to the one below:

Neo4j Bloom is accessible from inside Neo4j Desktop app.

Find out more information about to use Neo4j Bloom to explore your graph with:

Query the graph with Neo4j Browser

The Neo4j Browser lets you run Cypher queries on your graph to explore it and retrieve information from it. Cypher is like SQL for graphs, running Cypher queries inside the neo4j browser lets you explore the results as shown below:

The Neo4j Browser is available for both Enterprise and Community distributions. You can access it:

Visualize and explore with Linkurious Enterprise Explorer

Linkurious is a proprietary software which, similarly to Neo4j Bloom, lets you visualize and query your graph through a powerful UI.

Find out more information about Linkurious:

Visualize with Gephi

Gephi is a simple open-source visualization software. It is possible to export graphs from Datashare into the GraphML File Format and import them into Gephi.

Find out more information about:

Export your graph in the GraphML format

To export the graph in the GraphML file format, navigate to the 'Projects', select your project, choose the 'Graph ML' export format and click the 'Export graph' button:

In case you want to restrict the size of the exported graph, you can restrict the export to a subset of documents and their entities using the 'File types' and 'Project directory' filters.

You will now be able to visualize the graph using Gephi by opening the exported GraphML file in it.

Last updated

Datashare is an open source project by the International Consortium of Investigative Journalists