> For the complete documentation index, see [llms.txt](https://icij.gitbook.io/datashare/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://icij.gitbook.io/datashare/concepts/cli-stages/scanquery.md).

# SCANQUERY

The SCANQUERY stage queues the indexed files a search query selects, so the next stage extracts them again.

SCANQUERY searches an existing Elasticsearch index and pushes the **file path** of every document the query matches onto the queue of the next stage. Combined with `INDEX`, it re-extracts only those files, with the extraction options of the current run. It is the opposite of [SCANIDX](/datashare/concepts/cli-stages/scanidx.md), which marks paths so INDEX skips them.

A typical use is to run OCR again on documents with a language the first pass got wrong:

```bash
datashare stage run \
  --stages SCANQUERY,DEDUPLICATE,INDEX \
  --defaultProject my-project \
  --searchQuery "language:WELSH" \
  --ocrLanguage cym \
  --elasticsearchAddress http://elasticsearch:9200 \
  --queueType REDIS \
  --redisAddress redis://redis:6379
```

## Input and output

|               |                                                                                              |
| ------------- | -------------------------------------------------------------------------------------------- |
| Reads         | the Elasticsearch index named after `--defaultProject`                                       |
| Writes        | the queue of the next stage, by default `extract:queue:deduplicate` or `extract:queue:index` |
| Distributable | **No.** It scrolls the query results from end to end; a second copy duplicates the work      |

## Options that affect it

| Option                 | Default            | Effect                                                                                                         |
| ---------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------- |
| `--searchQuery`        | none, **required** | The documents to re-extract, as a Datashare query string or a JSON Elasticsearch clause.                       |
| `-P, --defaultProject` | `local-datashare`  | The index to search.                                                                                           |
| `--reportName`         | none               | Leave it unset. When set, INDEX skips the paths already recorded as extracted, so the run may extract nothing. |
| `--scrollSize`         | `1000`             | Documents per scroll page.                                                                                     |
| `--scroll`             | `60000ms`          | Elasticsearch scroll duration.                                                                                 |
| `--queueName`          | `extract:queue`    | Prefix of the queue it writes to. Must match the INDEX run.                                                    |

## How it selects files

```mermaid
flowchart LR
    ES[("existing index")] --> SQ["SCANQUERY<br/>--searchQuery"]
    SQ -- "paths of the matching files" --> DD["DEDUPLICATE"]
    DD --> IDX["INDEX"]
    IDX -- "re-extracted documents" --> ES
```

## Execution details

* **A query is mandatory.** Without `--searchQuery`, or with a blank one, the stage fails instead of re-extracting the whole index.
* **The next stage must read paths**, which means `DEDUPLICATE` or `INDEX`. Any other stage after it, such as `NLP`, is rejected.
* **It cannot run with `SCAN`.** Both feed INDEX with paths, so asking for both in the same `--stages` is rejected.
* **An embedded document queues its container.** The path of an attachment or an archive entry is the file on disk that holds it, so a container with several matching embedded documents is queued once per match. Add `DEDUPLICATE` to extract it only once; the stage logs a warning when it is missing.
* **It only queues paths.** It does not touch the report map, nor the documents themselves: the re-extraction is done by INDEX, which updates the existing documents rather than duplicating them.

## Failure and restart

SCANQUERY only reads the index, so re-running it is safe. If it is interrupted, run it again: files already re-extracted are extracted a second time, which costs time and nothing else.

## See also

* [SCANIDX](/datashare/concepts/cli-stages/scanidx.md), which does the opposite: marks indexed paths so INDEX skips them.
* [ENQUEUEIDX](/datashare/concepts/cli-stages/enqueueidx.md), which selects documents with a query too, but queues document ids for the stages after INDEX.
* [DEDUPLICATE](/datashare/concepts/cli-stages/deduplicate.md), to extract each container once.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://icij.gitbook.io/datashare/concepts/cli-stages/scanquery.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.