Part 0: The Problem(s)

I found myself needing to train a model to identify the topic of help desk tickets, but I had 4 serious problems to solve first.

The data is sensitive, and could not be moved outside the organization’s environment.
My company computer is a mid-range laptop with no GPU or CUDA capability.
The actual data is messy, which is the reason that I need to build this model in the first place.
I do not have much time to manually label the data. (I am just one guy after all…)

Part 1: Creating and Labelling “The Data”

I had no time to look at thousands of tickets with my own eyes and label them accordingly. But I had an idea… ChatGPT does a number of other tasks for me, why not this as well?

But due to the sensitive nature of the data, it was not possible for me to send any data outside the organization’s environment (including sending data to OpenAI). So using ChatGPT to label my tickets for me was out of the question, right?

Not exactly… I just realized I could not use the actual data; I would have to use fake data. But where would I get the fake data that is “real” enough to actually train an effective model?

Why not ChatGPT? I used few-shot prompting to generate several-hundred plausible tickets. Then I fed those tickets back to ChatGPT to generate 10 plausible variations of each ticket that have the same meaning as the original ticket. (I will call the original synthetic tickets the “seed” tickets and the generated variations “derived” tickets.)


items = [
  {
    "type": "Incident",
    "description": "User reports their computer is running extremely slow, especially when opening multiple applications. Advised user to restart their computer and checked for updates. Initiated a system cleanup. Advised on monitoring the situation and to report back if issues persist."
  },
  {
    "type": "Request",
    "description": "Employee requests installation of Adobe Creative Cloud for graphic design tasks. Verified the necessity with the employee's manager and confirmed license availability. Scheduled installation for the next available IT slot."
  },
  {
    "type": "Incident",
    "description": "User experiences intermittent internet disconnections throughout the workday. Checked network settings and updated network drivers. Recommended moving closer to the Wi-Fi router for a stronger signal and, if issue persists, will consider issuing a Wi-Fi extender."
  },
  
]

I fed each of the seed tickets back into the OpenAI API to get labels for each ticket. I used few-shot prompting again to create a prompt for each ticket and get the output label data as a JSON object. This was the prompt I used in this case:


def generate_prompt(ticket: str) -> str:
  return f"""
You are a call center supervisor and your duty is to identify the issue/request and system/software
that the following ticket is about, as well as if the ticket is a Request or an Incident.
Please return the solution as a JSON formatted object with the fields:
  - Issue/Request: The simplest way to phrase the reason for this ticket
  - Software/System: Name of the software or system that the ticket is about (or "unknown" if cannot be determined)
  - Type: [Incident|Request]

EXAMPLE 1:
Input:
  Ticket Description: "Employee has requested installation of MATLAB for data analysis purposes on their workstation. Verified project requirements with the team lead and confirmed MATLAB license availability. Escalating to Software Installation Team"
Output:
  {
    "Issue/Request": "Software Installation",
    "Software/System": "MATLAB",
    "Type": "Request"
  }

EXAMPLE 2:
Input:
  Ticket Description: "User reports Outlook crashes upon opening an email with a specific attachment. Attempted opening in safe mode – successful. Identified add-in conflict and disabled problematic add-in. User confirmed issue resolved."
Output:
  {
    "Issue/Request": "Outlook crashes upon opening an email",
    "Software/System": "Outlook",
    "Type": "Incident"
  }

--- Ticket: "{ticket}"
"""

As you can see in the prompt, I wanted the following data returned for each ticket description:

Issue/Request: The simplest way to phrase the reason for this ticket
Software/System: Name of the software or system that the ticket is about (or “unknown” if cannot be determined)
Type: Incident or Request

I applied the labeling data to the derived tickets for each seed ticket as well since they were meant to have the same meaning. (If I had run all of them through instead it would cost a lot more API credits for no good reason.)

To get even more training data, I also used several ticket templates where the software name was a placeholder. Then I gathered a list of common real pieces of software and saved a version of each template with each piece of software. This was in hopes that the model would be very good at determining which software a ticket was talking about.


ticket_templates = [
  {
    "description": "User reports SOFTWARE_X crashes unexpectedly when attempting to save files.\nPerformed initial diagnostics to check log files for error codes.\n\nEscalating to Software Maintenance Team",
    "issue/request": "SOFTWARE_X Crashing When Saving Files",
    "software/system": "SOFTWARE_X",
    "type": "Incident"
  },
  
]

software_types = [
  "Microsoft Office 365",
  "Slack",
  "Zoom",
  "Salesforce",
  
]

def create_fake_data():
  with open('processed_data.csv', 'a', newline='') as f:
    writer = csv.writer(f)
    for software in software_types:
      for template in ticket_templates:
        ticket = template['description'].replace('SOFTWARE_X', software)
        writer.writerow([
          template['type'],
          ticket,
          template['issue/request'].replace('SOFTWARE_X', software),
          software
        ])

Part 2: Training the Model

At this point, I had spent a bit of money on OpenAI API credits and I seemingly had training data. The problem I was trying to solve is not a traditional classification problem; it is actually a text generation problem. I did not know what the problems and software names were going to be ahead of time, so I could not make categories out of them. After a little bit of Google searching, I settled on fine-tuning google/flan-t5-base (a text2text generation model).

The input to the model is the “content” of the ticket (the description). And the output is the “Issue/Request” and the “Software/System” in the format: “Issue/Request: issue/request
Software/System: software/system”

(I initially tried to output a JSON object, but I got bad results. Also the ‘\n’ is a newline character.)

I was doing all this on my own computer with an RTX 3060, not my company computer. So I did have access to CUDA for training. I only had to train a few epochs (like 4 or 5) before the output was surprisingly good.

Part 3: The Moment of Truth

I had spent a little bit of time and money trying this experiment, but was it even going to work on the real tickets?

I uploaded it to Hugging Face here: KameronB/sitcc-t5-classifier (check it out and give it a like if you think this is interesting), that way I could download it to my company laptop.

To my surprise, it actually worked pretty well!


# inputs
input_text = [
  "The customer is getting the following error when using rSATS:\nERROR: 'Failed to connect'.\nI have tried restarting the application and the computer, but the issue persists.\nEscalating to Team",
  
]

# parsed outputs
{'Software/System': 'rSATS', 'Issue/Request': 'rSATS Failed to connect error'}

There were a few types of tickets that confused it, due to me forgetting to represent tickets that were similar to them in the fake training data. After one epoch of fine-tuning on my company laptop with a bit of real data added (which took 5 hours 💀) it is actually very good.

Part 4: We Did It

It seems that I was able to circumvent my hardware and data security limitations. I generated synthetic data that was good enough to the real thing to fine-tune a model for a computer that has limited hardware capability.

Let’s say I had tried to manually label the real data and then train the model using the CPU on my company laptop. That project would have taken forever… at least 30 hours or so for the training alone on CPU.

This is the first time I have tried something like this, and I believe it to be a success.

I tend to be in situations where I have to do a lot with few resources, rigid policy constraints, little computational power, or all three at once. Taking a pretrained model and using synthetic data to fine-tune it makes it good enough that it will require minimal fine-tuning on the actual data. It can also greatly increase the speed of labelling your actual data.

When I “fine-fine-tuned” the model on my company laptop, I was able to run the model on real data first to label it. Then I only had to go through and make adjustments to the ones that it got wrong, instead of labelling them all from scratch.

My model is on Hugging Face: https://huggingface.co/KameronB/sitcc-t5-classifier

Let me know what you think! I have several more ideas to try.

Blog