Why did we create Prophecies?

ICIJ is a non-profit leading outstanding investigations involving hundreds of reporters around the world. Each of our project leverage the power of data and technology to investigate issues of global importance. To be able to simultaneously publish a story in so many media, every single line of data we use must be fact checked. This is, as we say internally, "our secret sauce". Some of the datasets we produce rely on a very heavy effort to be verified, sometimes with up to 4 different rounds of verification.

In order to make sure this workflow is well organized and efficient, we created Prophecies.

How can I use Prophecies?

Prophecies is a collaborative tool that must be available online to its users. Therefore, to use Prophecies you must install it on your server, as web application.

What skills are needed to install Prophecies?

Prophecies is a Python application built at the top of the Django framework. To ease installation, we publish a Docker image of the application on Docker Hub. The image is built both for ARM64 and AMD64 which ensure a wider level of compatibility.

To install Prophecies, you will need to have a good understanding of how Docker and Docker Compose work and how they can be used. You will also need to have basic knowledge of how to setup a PostgreSQL database. In this guide, we will use a database directly installed in a Docker.

To report a bug, please open an issue on our Github. To maximize the chance of getting help from our team and our community, we advise you to be as precise as possible, while detailing:

• your Operating System (Mac, Windows or Linux)

• the version of your Operating System

• the version of Prophecies

• screenshots of your issue

If for confidentiality reasons you don't want to open an issue on Github, please write to prophecies@icij.org.

This page introduce the key concepts used by Prophecies.

Prerequisites

Before installing Prophecies, ensure you have the following tools installed:

• Docker: A platform for developing, shipping, and running applications inside isolated environments known as containers. Learn more about Docker.

• Docker Compose: A tool for defining and running multi-container Docker applications. It uses a YAML file to configure your application's services, making it easier to manage complex setups. Get started with Docker Compose.

These tools are essential for running Prophecies efficiently and are widely used in the development and deployment of modern applications.

Setup steps

1. Prepare the Working Directory

Place this docker-compose.yml file in your chosen directory. This will be the working directory for your Docker Compose setup:

version: '3'

volumes:
  shared_static: {}
  postgresql_data: {}

services:
  nginx:
      image: nginx:latest
      ports:
        - "9999:9999"
      volumes:
        - ./nginx.conf:/etc/nginx/conf.d/default.conf
        - shared_static:/static/:ro
      depends_on:
        - web

  web:
    image: icij/prophecies:0.5.9
    environment:
      - ALLOWED_HOSTS=0.0.0.0,localhost,web
      - CSRF_TRUSTED_ORIGINS=http://localhost:9999
      - DATABASE_URL=postgres://postgres:postgres@db/prophecies
      - DEBUG=false
      - DJANGO_SETTINGS_MODULE=prophecies.settings.production
      - DJANGO_ADMIN_LOGIN=true
      - PROPHECIES_APP_LOGIN_URL=/admin/login/?next=/
      - PORT=8008
      - WAIT_HOSTS_TIMEOUT=60
      - WAIT_HOSTS=db:5432
    volumes:
      - shared_static:/code/prophecies/run/static
    depends_on:
      - db
      - migration
      - collectstatic
    expose:
      - "8008"

  migration:
    image: icij/prophecies:0.5.9
    command: sh -c '/usr/bin/wait && poetry run python manage.py migrate --noinput'
    environment:
      - WAIT_HOSTS=db:5432
      - WAIT_HOSTS_TIMEOUT=60
      - DATABASE_URL=postgres://postgres:postgres@db/prophecies
    depends_on:
      - db

  collectstatic:
    image: icij/prophecies:0.5.9
    command: poetry run python manage.py collectstatic --noinput
    volumes:
      - shared_static:/code/prophecies/run/static

  db:
    image: postgres:16
    restart: always
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
      POSTGRES_DB: prophecies
    command: postgres -c shared_preload_libraries=pg_stat_statements -c 'pg_stat_statements.track=all'
    volumes:
      - postgresql_data:/var/lib/postgresql/data

2. Configure NGINX

You need to create an NGINX configuration file as referenced in your Docker Compose file (./nginx.conf). Ensure this file is correctly configured to serve your Prophecies. It should be set up to listen on port 9999 and serve static files from the /static/ directory:

upstream web {
  ip_hash;
  server web:8008;
}

server {

    location /static/ {
        autoindex on;
        alias /static/;
    }

    location / {
        proxy_pass http://web/;
    }
    
    listen 9999;
    server_name localhost;
}

3. Starting Services

In your working directory, run the following command:

docker compose up -d

This command will download the necessary images (if not already downloaded), create the defined volumes, and start the services as background processes.

3. Database Migrations and Static Files

The docker-compose.yml file specifies services for running database migrations (migration) and collecting static files so they can be served by NGINX (collectstatic). These services should automatically perform these tasks when you start Docker Compose.

4. Create Admin User

The database is currently empty, without user. To provision your first admin user, you must use the command line:

docker compose exec -it web make createsuperuser

5. Accessing the Application

Once all services are up and running, the application should be accessible via : http://localhost:9999

6. Monitoring and Logs

To monitor the status of your Docker containers and check for any issues, you can use the Docker Compose logs command:

docker compose logs -f

The -f flag will follow the log output, allowing you to see real-time logs as they are generated.

7. Stopping and Removing Services

To stop the services without removing them, you can run:

docker compose stop

If you need to stop and remove all the containers, networks, and volumes associated with your Docker Compose setup, use the down command:

docker compose down --volumes

Additional Notes

• This configuration doesn't include SSL setup for NGINX, which is recommended for production.

• Regularly check for updates to the icij/prophecies image to keep your application up to date.

• To use custom domain, don't forget to update the ALLOWED_HOSTS variable.

• The application can be configured with many environment variables.

Prerequisites

• Python 3.9

• Node 16.x

• Poetry >= 1.2

• Yarn 1.x

• Git

Setup steps

Checkout the repository with git:

git clone git@github.com:ICIJ/prophecies.git
cd prophecies/

After entering the directory, setup a virtualenv with poetry and to install required packages:

make install

To setup the database (using SQLite3 by default):

make migrate

To create a superuser:

make createsuperuser

For more customization, this app utilizes 12factor inspired environment variables to configure your Django application. You can create .env file using the custom settings variables:

DEBUG=on
DATABASE_URL=
CACHE_URL=dummycache://
STATIC_URL=/static/
SOCIAL_AUTH_PROVIDER_KEY=
SOCIAL_AUTH_PROVIDER_SECRET=
SOCIAL_AUTH_PROVIDER_HOSTNAME=http://localhost:3001
SOCIAL_AUTH_PROVIDER_USERNAME_FIELD=uid
SOCIAL_AUTH_PROVIDER_GROUPS_FIELD=groups_by_applications.prophecies
SOCIAL_AUTH_PROVIDER_STAFF_GROUP=icijstaff

The application can be configured with many environment variables.

The application can be configured using the following environment variables:

Once you , simply launch Pytest with this shortcut:

This command triggers the testing process, where make is a build automation tool used to manage tasks like compiling the code or running tests, and test-back is a target defined within the Makefile.

Upon execution, Pytest will begin to run all the specified tests in your test suite. You can monitor the output in your terminal to check the progress of the tests. If all goes well and your code passes all the tests, you should see a success message indicating that everything is working as intended. Any failed tests will be reported accordingly, allowing you to make the necessary adjustments.

Docker image tag

The Docker image is tagged with the version extracted from the git tag (e.g., icij/prophecies:1.0.0). Additionally, the image is tagged as latest.

• Image Repository:

• Platform: linux/arm64, linux/amd64

This documentation provides a comprehensive guide on how to release a new Docker image, build, and test the application locally using the provided Makefile commands.

Releasing a new Docker Image

1. Update Version

The version can be updated as major, minor, or patch using the Makefile. To do this, run one of the following commands depending on the type of version bump you need:

These commands automatically update the version in pyproject.toml for the Python backend and package.json in the frontend directory, then commit these changes and tag the version in Git.

2. Push the Changes and Tag to Git

After updating the version and creating a git tag, push the changes and the tag to your git repository:

Pushing the tag is crucial because the GitHub Actions workflow for Docker image publishing is triggered by a push to tags that follow the semantic versioning format, prefixed with v (e.g., v1.0.0).

Build and test locally

A convenience docker-compose.yml file is located as the root of the repository. To build and test Prophecies locally with Docker, you can run from the app's root directory:

Once you , simply launch Jest with this shortcut:

This command triggers the testing process, where make is a build automation tool used to manage tasks like compiling the code or running tests, and test-front is a target defined within the Makefile.

Upon execution, Jest will begin to run all the specified tests in your test suite. You can monitor the output in your terminal to check the progress of the tests. If all goes well and your code passes all the tests, you should see a success message indicating that everything is working as intended. Any failed tests will be reported accordingly, allowing you to make the necessary adjustments.

Mock Service Worker

This application utilizes (MSW) to mock API results from the backend. All server request are defined within handers that you can find :

is designed to automate the process of linting, testing, and publishing Docker images. This workflow triggers on every push to the repository and is structured into several jobs to ensure code quality and reliability before a new Docker image is published.

Jobs Overview

The workflow consists of six main jobs.

lint-backend

• Purpose: Lints the backend code to ensure it adheres to Python coding standards.

• Python Version: Runs with Python 3.9.

• Command: Uses pylint to lint the prophecies directory.

test-backend

• Purpose: Executes backend tests to verify the functionality.

• Python Version: Tests are run on both Python 3.9 and 3.10.

• Command: Runs backend tests using the make test-back command.

test-frontend

• Purpose: Conducts frontend tests to ensure UI/UX integrity.

• Node Version: Tests are conducted on Node.js versions 16.x and 18.x.

• Command: Frontend tests are executed using the make test-front command.

build

• Purpose: Build artifacts to be publish with the next release.

• Conditions: This job runs only if the push event is a tag push starting with 'v', and it depends on the successful completion of the lint-backend, test-backend, and test-frontend jobs.

• Command: Run a python command to extract the OpenAPI schema file and store it.

package-publish

• Purpose: Restore artifacts from the build job to publish them as Github Release.

• Conditions: This job runs only if the push event is a tag push starting with 'v', and it depends on the successful completion of the build job.

docker-publish

• Conditions: This job runs only if the push event is a tag push starting with 'v', and it depends on the successful completion of the build job.

• Step: This job includes checking out the code, setting up QEMU and Docker Buildx, logging into Docker Hub, preparing the tag name based on the git tag, and finally building and pushing the Docker image.

Secrets

To be able to push on the Docker Hub, this workflow must use the following secrets:

• DOCKERHUB_USERNAME: The username for Docker Hub authentication.

• DOCKERHUB_PASSWORD: The password or access token for Docker Hub authentication.

API

The API is built with Django Rest Framework and implements the of Prophecies.

Stack

Prophecies is based on the following open source tools:

• Python 3.9, 3.10

• Node 16, 18

• Django 4.2

• Django Rest Framework 3.14

• Django Rest Framework JSON:API 6.1

• Pytest

• Vue 2.7

• Vue CLI 5

• Vuex ORM 0.36

• Jest 28

• Poetry

• Yarn

Interaction workflow

This sequence diagram illustrates the communication flow within the Prophecies platform infrastructure, focusing on how a request travels from the user interface to the database and back.

Database schema

Here is a simplified version of the Database schema.