> For the complete documentation index, see [llms.txt](https://icij.gitbook.io/datashare/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://icij.gitbook.io/datashare/concepts.md).

# Concepts

- [Running modes](https://icij.gitbook.io/datashare/concepts/running-modes.md): Datashare runs using different modes with their own features.
- [CLI stages](https://icij.gitbook.io/datashare/concepts/cli-stages.md): When running Datashare from the command-line, pick which "stages" to apply to process your documents.
- [SCAN](https://icij.gitbook.io/datashare/concepts/cli-stages/scan.md): The SCAN stage walks a directory and queues every file it finds.
- [SCANIDX](https://icij.gitbook.io/datashare/concepts/cli-stages/scanidx.md): The SCANIDX stage records the paths already present in an index, so a later run can skip them.
- [DEDUPLICATE](https://icij.gitbook.io/datashare/concepts/cli-stages/deduplicate.md): The DEDUPLICATE stage filters paths that appear more than once in the queue during a run.
- [INDEX](https://icij.gitbook.io/datashare/concepts/cli-stages/index.md): The INDEX stage extracts text, metadata and embedded documents, and writes them to Elasticsearch. It is the expensive one.
- [ENQUEUEIDX](https://icij.gitbook.io/datashare/concepts/cli-stages/enqueueidx.md): The ENQUEUEIDX stage pushes documents that are already indexed back onto a queue, so a later stage can process them.
- [CATEGORIZE](https://icij.gitbook.io/datashare/concepts/cli-stages/categorize.md): The CATEGORIZE stage fills the contentTypeCategory field of documents that are already indexed.
- [NLP](https://icij.gitbook.io/datashare/concepts/cli-stages/nlp.md): The NLP stage extracts named entities from documents that are already indexed.
- [CREATENLPBATCHESFROMIDX](https://icij.gitbook.io/datashare/concepts/cli-stages/createnlpbatchesfromidx.md): The CREATENLPBATCHESFROMIDX stage groups indexed documents into batches and submits one named entity recognition task per batch.
- [BATCHNLP](https://icij.gitbook.io/datashare/concepts/cli-stages/batchnlp.md): BATCHNLP exists in the stage list but is not runnable from the command line.
- [ARTIFACT](https://icij.gitbook.io/datashare/concepts/cli-stages/artifact.md): The ARTIFACT stage caches the payload of embedded documents on disk, so previews and downloads do not re-parse their container.
- [CLI reference](https://icij.gitbook.io/datashare/concepts/cli-reference.md): Complete reference of Datashare's command-line interface subcommands and global flags.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://icij.gitbook.io/datashare/concepts.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.