> For the complete documentation index, see [llms.txt](https://icij.gitbook.io/datashare/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://icij.gitbook.io/datashare/concepts/cli-stages/enqueueidx.md).

# ENQUEUEIDX

The ENQUEUEIDX stage pushes documents that are already indexed back onto a queue, so a later stage can process them.

ENQUEUEIDX scrolls an Elasticsearch index and pushes the id of every matching document onto the queue of the next stage. It is the entry point of every "do something to documents I already have" pipeline, and it is always used in front of another stage.

```bash
datashare stage run \
  --stages ENQUEUEIDX,NLP \
  --defaultProject my-project \
  --nlpPipeline CORENLP \
  --elasticsearchAddress http://elasticsearch:9200 \
  --queueType REDIS \
  --redisAddress redis://redis:6379
```

## Input and output

|               |                                                                                          |
| ------------- | ---------------------------------------------------------------------------------------- |
| Reads         | the Elasticsearch index named after `--defaultProject`                                   |
| Writes        | the queue of the next stage, for example `extract:queue:nlp` or `extract:queue:artifact` |
| Distributable | **No.** It scrolls the index from end to end; a second copy enqueues everything twice    |

## Options that affect it

| Option                 | Default           | Effect                                                                                     |
| ---------------------- | ----------------- | ------------------------------------------------------------------------------------------ |
| `-P, --defaultProject` | `local-datashare` | The index to read.                                                                         |
| `--searchQuery`        | none              | Restricts what is enqueued. Datashare query syntax, or a raw Elasticsearch clause in JSON. |
| `--nlpPipeline`        | `CORENLP`         | Used by the default selection when the next stage is NLP, see below.                       |
| `--scrollSize`         | `1000`            | Documents per scroll page.                                                                 |
| `--scroll`             | `60000ms`         | Elasticsearch scroll duration.                                                             |

## Execution details

* **What it selects depends on the stage that follows it.** With no `--searchQuery`:
  * in front of `NLP`, it selects the documents **not yet processed** by `--nlpPipeline`. This is what makes an interrupted NER run resumable: rerun the same command and it picks up where it stopped;
  * in front of any other stage, it selects every document in the index.
* **`--searchQuery` replaces that selection entirely.** With a query, the "not yet processed" filter no longer applies, so an `ENQUEUEIDX,NLP` run with a query will re-process documents the pipeline has already seen.
* **Two query syntaxes are accepted.** A value starting with `{` and ending with `}` is treated as a raw Elasticsearch clause and used as the `must` clause of a bool query, so pass the clause itself rather than a full `{"query": ...}` body. Anything else is parsed with the same syntax as the Datashare search bar.

  ```bash
  --searchQuery 'language:FRENCH'
  --searchQuery '{"term":{"contentType":"application/pdf"}}'
  ```
* **It enqueues ids, not content.** The document body is fetched again by the consuming stage, which is why this stage is cheap even on a very large index.
* **Results are sorted by language.** That is what lets the NLP stages group work by language and avoid reloading a model for every document.
* **Embedded documents are included** like any other document. To enqueue only the files that came from disk, filter on `extractionLevel:0`, which is what the ARTIFACT stage wants.

## Failure and restart

The stage keeps no state. If it is interrupted, rerun it: in front of NLP the "not yet processed" filter means it naturally enqueues only what remains, and elsewhere it re-enqueues everything, which costs a second pass but produces the same result.

## See also

* [NLP](/datashare/concepts/cli-stages/nlp.md), [CATEGORIZE](/datashare/concepts/cli-stages/categorize.md), [ARTIFACT](/datashare/concepts/cli-stages/artifact.md) and [LANGUAGE](/datashare/concepts/cli-stages/language.md), the stages usually placed after it.
* [Scenario 7: extract named entities after indexing](/datashare/server-mode/indexing/scenarios.md#scenario-7-extract-named-entities-after-indexing)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://icij.gitbook.io/datashare/concepts/cli-stages/enqueueidx.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.