Optimizing AI for Domain-Specific Tasks: The Case of GPT-4o-mini in Hospitality

Introduction

Recently, I published a detailed article on Medium exploring how fine-tuning GPT-4o-mini could dramatically improve its performance on hospitality-specific tasks. This companion piece highlights key insights from that project while approaching the topic from a different angle, focusing on the broader implications for domain adaptation in language models.

My original experiment demonstrated that a fine-tuned GPT-4o-mini model could achieve 60% accuracy on hospitality intent classification—surpassing even GPT-4.1's 52%—while maintaining substantially lower costs and better response times.

The Challenge of Nuanced Communication in Hospitality

The hospitality industry presents unique natural language understanding challenges. Hotel guests not always express their needs in straightforward terms. Consider these examples:

"I need to attend an online meeting after my official departure time." "Our flight doesn't leave until evening, but checkout is at noon." "I was hoping to use the facilities before heading to the airport."

All three statements indirectly request late checkout, but their linguistic structure varies significantly. Traditional chatbots struggle with this indirectness, leading to frustrating guest experiences.

My research tackled this challenge by fine-tuning language models to better recognize 40 common hospitality intents hiding behind ambiguous phrasing. The project demonstrated that a specialized model can far outperform general-purpose models at this task, even when the specialized model is significantly smaller.

Dataset Creation for Domain Adaptation

The foundation of successful domain adaptation is high-quality training data. For this project, I created two specialized datasets:

• A training set of 400 ambiguous hospitality requests paired with their correct intent classifications
• An evaluation set of 100 previously unseen examples for benchmarking

To support further research in this area, I've published both datasets on HuggingFace as part of the Hospitality Intent Classification Challenge collection:

• VagueIntent-Train (400 samples)
• VagueIntent-Eval (100 samples)

These datasets address a practical gap in hospitality AI development: the need for standardized data to train and evaluate models on understanding indirect guest requests.

Each example in the dataset follows this structure:

{
  "messages": [
    {
      "role": "developer",
      "content": "You are an advanced hospitality chatbot for a premium hotel chain..."
    },
    {
      "role": "user",
      "content": "I won't be departing until later tonight, but I notice checkout is at 11 AM."
    },
    {
      "role": "assistant",
      "content": "7"
    }
  ]
}

Where "7" represents the numerical code for "Request late check-out" in our classification system.

Fine-tuning Methodology

The fine-tuning process leveraged OpenAI's API to adapt the GPT-4o-mini model to this classification task. Key technical decisions included:

• Limiting to a single epoch to prevent overfitting
• Setting learning rate multiplier to 0.3
• Using "auto" batch size optimization

This approached proved to be quite effective. My Medium article details the technical implementation for those interested in the specifics.

What's particularly notable is that these optimization choices seem to effectively balance model adaptation while preserving general capabilities. The model became a hospitality specialist without compromising its underlying language understanding.

Performance Comparisons

The evaluation revealed striking differences in model capability:

The dramatic improvement—from 26% to 60% accuracy—demonstrates how effective domain-specific adaptation can be, even with relatively small datasets. What's more surprising is that the smaller fine-tuned model outperformed the much larger GPT-4.1 model by a significant margin.

Technical Limitations and Workarounds

While the overall results were impressive, I encountered several technical limitations:

DPO Fine-tuning Constraints

OpenAI's documentation suggests using Supervised Fine-Tuning (SFT) first, followed by Direct Preference Optimization (DPO). However, I discovered that the API currently only supports DPO with base models, not models that have already undergone SFT. This prevented implementing the full two-stage optimization process.

Documentation Inconsistencies

Throughout the project, I encountered discrepancies between OpenAI's documentation and API behavior. The API calls in their guides didn't always match the expected format. Fortunately, most of these issues have been addressed in recent documentation updates.

Key Insights for Domain Adaptation

This project reinforced several important principles for domain adaptation:

1. Specialized Beats General

The most powerful insight is that specialized models consistently outperform general models on domain-specific tasks. A relatively small model with domain expertise can outperform a much larger general model.

2. Data Quality Trumps Quantity

With just 400 training examples, we achieved remarkable improvement. This underscores that carefully constructed, high-quality data focused on the specific domain challenge is more valuable than massive quantities of general data.

3. Cost Efficiency at Scale

For organizations handling thousands or millions of similar queries, the economics of specialized models become increasingly compelling. The initial investment in fine-tuning pays dividends through both improved performance and reduced operational costs.

4. Start Small, Then Optimize

The project demonstrated that even a single training epoch with minimal hyperparameter tuning can yield impressive results. This suggests that organizations should start with simple fine-tuning approaches and only invest in more complex optimization when necessary.

Future Research Directions

This project opens several promising research avenues:

Multi-Intent Recognition

Many guest requests contain multiple intents ("I'd like to check out late and arrange for airport transportation"). Future work could focus on recognizing and prioritizing multiple intents within a single guest request.

Comparative Studies Across Model Families

Research comparing the effectiveness of fine-tuning across different model families (GPT, Claude, Llama, Mistral, etc.) would provide valuable insights into which models are most adaptable to domain-specific tasks.

Dataset Size

OpenAPI guide for fine-tuning says:

To fine-tune a model, you are required to provide at least 10 examples. We typically see clear improvements from fine-tuning on 50 to 100 training examples with gpt-4o-mini and gpt-3.5-turbo, but the right number varies greatly based on the exact use case. We recommend starting with 50 well-crafted demonstrations and seeing if the model shows signs of improvement after fine-tuning.

In my project I initially used 200 test samples and got 56% for accuracy.

With 400 test samples I got 60%.

And this is specifically for vague queries. If queries are not very vague, gpt-4o-mini respond with correct message.

With more test samples for fine-tuning we can expect accuracy should increase.

Conclusion: Making the Most of Smaller Models

The hospitality intent classification project demonstrates a key principle for practical AI applications: specialized, domain-adapted smaller models can outperform larger general-purpose models while offering significant cost advantages. For organizations implementing AI solutions in specific domains, this suggests a clear strategy:

1. Identify the specific language understanding challenges in your domain
2. Create focused training datasets addressing those challenges
3. Fine-tune smaller models rather than defaulting to larger models
4. Continuously evaluate and refine based on real-world performance

By following this approach, organizations can achieve both better performance and lower costs—a rare win-win in the world of AI implementation.

Have you experimented with domain adaptation for smaller models? Share your experiences in the comments.

The complete code for the OpenAI portion of this project is available on GitHub, and the datasets are accessible on HuggingFace.