README.md

---
license: cc-by-nc-4.0
---


This model is pretrained from the checkpoint of [`xlnet-base-cased`](https://huggingface.co/xlnet-base-cased) for the mental healthcare domain. XLNet model pre-trained on English language. It was introduced in the paper [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Yang et al. and first released in [this repository](https://github.com/zihangdai/xlnet/). 


## Usage

Here is how to use this model to get the features of a given text in PyTorch:

```python
from transformers import XLNetTokenizer, XLNetModel

tokenizer = XLNetTokenizer.from_pretrained('AIMH/mental-xlnet-base-cased')
model = XLNetModel.from_pretrained('AIMH/mental-xlnet-base-cased')

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state
```

To minimize the influence of worrying mask predictions, this model is gated.  To download a gated model, you’ll need to be authenticated. 
Know more about [gated models](https://huggingface.co/docs/hub/models-gated). 

**This model is biased due to training with posts about self-reported mental conditions and should not be used for text generation application, e.g., chatbot for mental health counseling.** 

## Paper
```
@article{ji-domain-specific,
  author        = {Shaoxiong Ji and Tianlin Zhang and Kailai Yang and Sophia Ananiadou and Erik Cambria and J{\"o}rg Tiedemann},
  journal       = {arXiv preprint arXiv:2304.10447},
  title         = {Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health},
  year          = {2023},
  url           = {https://arxiv.org/abs/2304.10447}
}
```

## Disclaimer

The model predictions are not psychiatric diagnoses.
We recommend anyone who suffers from mental health issues to call the local mental health helpline and seek professional help if possible.