> For the complete documentation index, see [llms.txt](https://icij.gitbook.io/datashare/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://icij.gitbook.io/datashare/concepts/cli-stages/categorize.md).

# CATEGORIZE

The CATEGORIZE stage fills the contentTypeCategory field of documents that are already indexed.

CATEGORIZE pops document ids from its queue, reads each document's content type from Elasticsearch, derives a coarse category from it, and updates the document. That category is what the file type filter in the interface uses.

It is a backfill stage: documents indexed by a recent version already carry the field.

```bash
datashare stage run \
  --stages ENQUEUEIDX,CATEGORIZE \
  --defaultProject my-project \
  --elasticsearchAddress http://elasticsearch:9200 \
  --queueType REDIS \
  --redisAddress redis://redis:6379
```

## Input and output

|               |                                                                                                       |
| ------------- | ----------------------------------------------------------------------------------------------------- |
| Reads         | `<queueName>:categorize`, and documents from Elasticsearch                                            |
| Writes        | the `contentTypeCategory` field back to Elasticsearch, and forwards each id to the next stage's queue |
| Distributable | **Yes.** Several machines can drain the same queue                                                    |

## Options that affect it

| Option                        | Default                    | Effect                               |
| ----------------------------- | -------------------------- | ------------------------------------ |
| `-P, --defaultProject`        | `local-datashare`          | The index to read from and write to. |
| `--queueName` / `--queueType` | `extract:queue` / `MEMORY` | The queue it drains.                 |

It has no options of its own. What gets categorized is decided upstream by [ENQUEUEIDX](/datashare/concepts/cli-stages/enqueueidx.md), including its `--searchQuery`.

## Execution details

* **The loop is single-threaded**, one document at a time. Parallelism comes from running the stage in several processes.
* **The category is derived from the content type already stored in the index.** Nothing is re-parsed and no file is re-read, which makes this the cheapest stage in the pipeline.
* **A document with no content type gets the `OTHER` category** rather than being skipped.
* **It forwards ids to the next stage.** Unlike NLP, CATEGORIZE passes each entry on, so `ENQUEUEIDX,CATEGORIZE,NLP` runs both over the same set of documents in one command.
* **A missing document is skipped with a warning**, and a failure on one document is logged without stopping the loop.
* **It is idempotent.** Recomputing the category of a document that already has one writes the same value.

{% hint style="info" %}
With the Temporal task manager, this stage has a 10 minute activity timeout, which is short compared to the other stages. Enqueue in batches if you are backfilling a very large index on that setup.
{% endhint %}

## See also

* [ENQUEUEIDX](/datashare/concepts/cli-stages/enqueueidx.md), which feeds this stage.
* [Scenario 10: backfills on an existing index](/datashare/server-mode/indexing/scenarios.md#scenario-10-backfills-on-an-existing-index)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://icij.gitbook.io/datashare/concepts/cli-stages/categorize.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.