The advent of generative AI marked a pivotal moment in the technology landscape, particularly with the introduction of ChatGPT in November 2022. This breakthrough triggered an unprecedented surge of interest in artificial intelligence, extending far beyond the tech-savvy elite to enterprises across various sectors. Large language models (LLMs), known for their capacity to understand and generate human-like text, quickly became a focal point for businesses aiming to enhance productivity and streamline operations. The initial wave of enthusiasm led to the development of numerous AI-driven tools designed to reduce manual tasks, such as AI data analysts, automated insight generators, and knowledge search functionalities. However, as enterprises ventured further into the realm of AI, the challenges of integrating these technologies into real-world business applications became increasingly apparent.

Building an AI assistant capable of answering questions based on structured and unstructured data is a relatively straightforward task in today’s technologically advanced world. Yet, these assistants often struggle with more nuanced or complex business questions, which require a deeper understanding of context and an ability to reason across multiple data points. For instance, consider an AI assistant equipped with detailed information on sales, segmented by region, brand, sales channel, and date. While such a system might excel in answering basic queries, it often falters when faced with more intricate business challenges that demand a higher level of contextual awareness.

Challenges in Developing Enterprise LLM Applications

The deployment of LLMs in enterprises reveals a spectrum of challenges, ranging from the relatively simple to the exceedingly complex. These challenges are best understood through a series of examples, each highlighting a different level of difficulty in question answering and the corresponding strategies for overcoming these hurdles.

Easy to Answer Reliably: Simple queries, such as “What are the sales through various channels?” or “Which region has the highest sales?” pose no significant challenge for LLMs. These questions rely on straightforward data retrieval and do not require any complex reasoning. Therefore, LLMs can handle them with ease, delivering accurate and reliable answers.

Slightly Difficult, Solvable with Engineering: Questions such as “How are my XYZ chocolate sales trending?” or “What were the incremental sales during the holiday campaign?” introduce a layer of complexity. Here, the LLM needs to recognize that XYZ Chocolates is a brand within the dataset and understand the concept of incremental sales within the context of a specific holiday campaign. These challenges can be addressed by providing the LLM with a semantic context layer—an additional information layer that includes brand names, holiday calendars, and metric definitions. This context layer enables the LLM to generate more accurate and contextually relevant responses.

More Complex, Requiring Decision Aid: When faced with questions like “What is the impact on sales due to an increase in inflation?” the LLM encounters a more significant challenge. To answer such a question, the LLM must understand the relationship between inflation and sales and have access to specific data on inflation rates. A possible solution involves using a machine learning model to establish the correlation between these variables, coupled with user input to refine the analysis. This approach allows the LLM to provide a more nuanced and informed answer.

Strategic Questions with Multiple Solutions: The most challenging category involves strategic questions, such as “How can I increase sales by 5%?” These queries require the LLM to consider numerous data points and apply judgment, a task that is traditionally within the realm of human decision-makers. While LLMs are not yet capable of fully replicating human reasoning, they can be guided to provide useful insights by constraining the problem and offering a limited set of options. For example, if the LLM has access to region-wise demand forecasts and marketing spend models, it can suggest optimizing marketing expenditures in regions with low demand as a strategy to boost sales.

From these examples, several key patterns emerge that are crucial for designing reliable LLM assistants capable of providing contextual answers. One of the most significant is the creation of a Semantic Context Layer—an information layer that helps the LLM understand business nuances, such as table and column descriptions, a glossary of terms, detailed data catalogues, metric definitions, user personas, historical SQL queries, and the relationships between different data tables.

The Role of Worker Agents in Enhancing LLM Applications

In addition to the semantic context layer, the development of Worker Agents plays a crucial role in the functionality of LLMs within enterprise environments. Worker agents are specialized tools that utilize enterprise data to perform specific tasks with a high degree of accuracy. These agents, which can range in complexity, often serve as reusable organizational assets that provide valuable intelligence to LLM assistants.

Examples of worker agents include:

• Reusable Codebase: Predefined logic for tasks such as customer prioritization or methods for estimating incremental sales.
• Models: Predictive models that establish relationships between key drivers and target key performance indicators (KPIs) like sales, enabling the LLM to answer “what if” scenarios.
• LLM Agents: Text-to-SQL generators or Retrieval-Augmented Generation (RAG) analysis tools that excel at performing specific tasks, such as querying databases or summarizing documents.
• Dashboards: Visual summaries of data insights that the AI can interpret to answer complex questions.

At the heart of this system is the LLM Brain, which orchestrates the entire process from question interpretation to answer generation. The LLM Brain performs several critical functions, including:

• Interpreting the Question: Understanding the intent behind the user’s query.
• Seeking User Input: Engaging the user when additional information or clarification is required.
• Accessing the Context Layer: Enriching the query with relevant business context.
• Deciding on Worker Agents: Determining which worker agents are needed to generate an accurate answer.
• Summarizing the Output: Presenting the answer in a human-readable format that is easy to understand.
• Validating the Output: Ensuring that the generated response is free from sensitive or harmful content.

This comprehensive approach to LLM development has been successfully applied in various real-world applications. For instance, an AI assistant designed for pharmaceutical representatives can help plan meetings with physicians by analyzing past interactions and prescribing patterns. Similarly, an AI assistant for data users can sift through thousands of enterprise datasets to identify the most relevant information for a given query. Another example is an AI assistant that helps marketing teams plan their budgets by generating spending scenarios based on historical data and predictive models.

Challenges and the Way Forward in AI Development

While the approach of contextualizing AI and employing worker agents holds immense promise for transforming business operations, several challenges remain. Achieving analysis and reasoning skills on par with human experts is one of the most significant obstacles. A primary challenge lies in the lack of comprehensive business context. Building and maintaining a robust context layer is a time-consuming process, but it is essential for effective AI assistance. Without a deep understanding of the business environment, AI assistants are limited in their ability to provide meaningful insights.

Moreover, while LLMs continue to improve, their ability to reason and coordinate multiple agents remains constrained. There is a current upper limit to the complexity these models can handle, although this limit is rapidly expanding. To address these limitations, ongoing research is focusing on the development of multi-agent systems where individual agents possess enhanced reasoning capabilities and can collaborate to iteratively solve complex problems.

Another significant obstacle is the existence of data silos and disparate systems within enterprises. Integrating these diverse data sources is crucial for creating AI assistants that can answer a broad range of questions. However, the process of data integration is often fraught with challenges, including issues related to data quality, consistency, and governance.

Despite these challenges, the rapid evolution of AI models is driving progress in both accuracy and efficiency. Domain-specific model training is becoming increasingly important as AI assistants are tasked with tackling more complex and industry-specific problems. Custom business assistants that are tailored to an organization’s unique needs, as opposed to off-the-shelf solutions, have the potential to significantly enhance business productivity, efficiency, and security.

The Future of AI in Enterprise

The future of AI in enterprise settings is bright, with the potential to revolutionize various aspects of business operations. The contextualization and multi-agent approach of organizational intelligence promises to unlock unprecedented levels of enterprise intelligence. Applications of these advanced AI systems are vast, spanning customer service, supply chain management, human resources, sales, and marketing. While these areas have already seen significant benefits from advanced analytics, the integration of LLMs and worker agents has the potential to take these capabilities to the next level.

As AI continues to evolve, organizations that invest in the development of custom AI assistants with deep organizational insights will be better positioned to capitalize on the transformative potential of this technology. However, realizing this potential will require ongoing innovation, model refinement, and a commitment to overcoming the challenges associated with building and maintaining robust AI systems. By addressing these challenges head-on, enterprises can harness the full power of AI to drive growth, improve decision-making, and stay ahead of the competition in an increasingly digital world.