# Data Preprocessing

By default, the project removes **phone numbers, ID numbers, emails, and URLs** from the data.\
In the `settings.jsonc` file, there is also a **blocked words list** (`blocked_words`) where you can add specific words or phrases to filter out. (Entire sentences containing blocked words will be removed by default.)

> **Important**\
> 🚨 *Please make sure to protect personal privacy and do not leak any personal information!*

To preprocess your data, run the following command:

```
weclone-cli make-dataset
```

You can modify `make_dataset_args` in `settings.jsonc` to match your chat style.

Currently, only the **time-window strategy** is supported:

* Messages from the same person within a certain time (`single_combine_time_window`) will be merged into a single sentence using commas
* Q\&A pairs will be matched based on `qa_match_time_window`

You can enable `enable_clean` under `clean_dataset` to clean your data for better results.

The system currently supports **LLM-based scoring** of chat records using either:

* **Offline inference** via `vllm`, or
* **Online API-based inference**

To enable API-based inference, set `"online_llm_clear": true` in `settings.jsonc` and configure:

* `base_url`
* `llm_api_key`
* `model_name`

All models compatible with the **OpenAI API interface** can be used.

Once you have the **distribution of LLM scores**, you can use the `accept_score` parameter to filter acceptable score ranges. You may also **lower the `lora_dropout`** parameter under `train_sft_args` to improve the model’s ability to fit the data.