This paper surveys the evolution of generative AI, highlighting innovations in MoE, multimodality, and AGI while addressing ethical and research challenges.
Abstract—This comprehensive survey explored the evolving landscape of generative Artificial Intelligence (AI), with a specific focus on the transformative impacts of Mixture of Experts (MoE), multimodal learning, and the speculated advancements towards Artificial General Intelligence (AGI). It critically examined the current state and future trajectory of generative Artificial Intelligence (AI), exploring how innovations like Google’s Gemini and the anticipated OpenAI Q* project are reshaping research h priorities and applications across various domains, including an impact analysis on the generative AI research taxonomy. It assessed the computational challenges, scalability, and realworld implications of these technologies while highlighting their potential in driving significant progress in fields like healthcare, finance, and education. It also addressed the emerging academic challenges posed by the proliferation of both AI-themed and AIgenerated preprints, examining their impact on the peer-review process and scholarly communication. The study highlighted the importance of incorporating ethical and human-centric methods in AI development, ensuring alignment with societal norms and welfare, and outlined a strategy for future AI research that focuses on a balanced and conscientious use of MoE, multimodality, and AGI in generative AI.
**Index Terms—AI Ethics, Artificial General Intelligence (AGI), Artificial Intelligence (AI), Gemini, Generative AI, Mixture of Experts (MoE), Multimodality, Q* (Q-star), Research Impac t Analysis.
I. INTRODUCTION
THE historical context of AI, tracing back to Alan Turing’s “Imitation Game” [1], early computational theories [2], [3], and the development of the first neural networks and machine learning [4], [5], [6], has set the foundation for to - day’s advanced models. This evolution, accentuated by crucial moments such as the rise of deep learning and reinforcement learning, has been vital in shaping the contemporary trends in AI, including the sophisticated Mixture of Experts (MoE) models and multimodal AI systems, illustrating the field’s dynamic and continuously evolving character. These advancements are a testament to the dynamic and ever-evolving nature of AI technology. The evolution of Artificial Intelligence (AI) has witnessed a crucial turn with the advent of Large Language Models (LLMs), notably ChatGPT, developed by OpenAI, and the recent unveiling of Google’s Gemini [7], [8]. This technology has not only revolutionized the industry and academia but has also reignited critical discussions concerning AI consciousness and its potential threats to humanity [9], [10], [11]. The development of such advanced AI systems, including notable competitors like Anthropic’s Claude, and now Gemini, which demonstrates several advances over previous models like GPT-3 and Google’s own LaMDA, has reshaped the research landscape. Gemini’s ability to learn from two-way conversations and its “spike-and-slab” attention method, which allows it to focus on relevant parts of the context during multi-turn conversations, represents a significant leap in developing models that are better equipped for multidomain conversational applications [1]. These innovations in LLMs, including the mixture-of-experts methods employe d by Gemini, signal a move towards models that can handle a diversity of inputs and foster multimodal approaches. Amidst this backdrop, speculations of an OpenAI project known as Q* (Q-Star) have surfaced, allegedly combining the power of LLMs with sophisticated algorithms such as Q-learning and A* (A-Star algorithm), further contributing to the dynamic research environment [2].
A. Changing AI Research Popularity
As the field of LLMs continues to evolve, exemplified by innovations such as Gemini and Q*, a multitude of studies have surfaced with the aim of charting future research paths, which have varied from identifying emerging trends t o highlighting areas poised for swift progress. The dichotom y of established methods and early adoption is evident, with “hot topics” in LLM research increasingly shifting towards multimodal capabilities and conversation-driven learning, as demonstrated by Gemini. The propagation of preprints has expedited knowledge sharing, but also brings the risk of reduced academic scrutiny. Issues like inherent biases, noted by Retraction Watch, along with concerns about plagiarism and forgery, present substantial hurdles [12]. The academic world, therefore, stands at an intersection, necessitating a unified drive to refine research directions in light of the fast-paced evolution of the field, which appears to be partly traced through the changing popularity of various research keywords over time. The release of generative models like GPT and the widespread commercial success of ChatGPT have been influential. As depicted in Figure 1, the rise and fall of certain keywords appear to have correlated with significant industry milestones, such as the release of the “Transformer” model in 2017 [13], the GPT model in 2018 [14], and the commercial ChatGPT-3.5 in December 2022. For instance, the spike in searches related to “Deep Learning” coincides with the breakthroughs in neural network applications, while the interest in “Natural Language Processing” surges as models like GPT and LLaMA redefine what’s possible in language understanding and generation. The enduring attention to “Ethics / Ethical” in AI research, despite some fluctuations, reflects the continuous and deep-rooted concern for the moral dimensions of AI, underscoring that ethical considerations are not merely a reactionary measure, but an integral and persistent dialogue within the AI discussion [15].
It is academically intriguing to postulate whether these trends signify a causal relationship, where technological advancements drive research focus, or if the burgeoning research itself propels technological development. This paper also explores the profound societal and economic impacts of AI advancements. We examine how AI technologies are reshaping various industries, altering employment landscapes, and influencing socio-economic structures. This analysis highlights both the opportunities and challenges posed by AI in the modern world, emphasizing its role in driving innovation and economic growth, while also considering the ethical implications and potential for societal disruption. Future studies could yield more definitive insights, yet the synchronous interplay between innovation and academic curiosity remains a hallmark of AI’s progress.
Meanwhile, the exponential increase in the number of preprints posted on arXiv under the Computer Science > Artificial Intelligence (cs.AI) category, as illustrated in Figure 2, appears to signify a paradigm shift in research dissemination within the AI community. While the rapid distribution of findings enables swift knowledge exchange, it also raises concerns regarding the validation of information. The surge in preprints may lead to the propagation of unvalidated or biased information, as these studies do not undergo the rigorous scrutiny and potential retraction typical of peer-reviewed publications [16], [17]. This trend underlines the need for careful consideration and critique in the academic community, especially given the potential for such unvetted studies to be cited and their findings propagated.
B. Objectives
The impetus for this investigation is the official unveiling of Gemini and the speculative discourse surrounding Q* project, which prompts a timely examination of the prevailing currents in generative AI research. This paper specifically contributes to the understanding of how MoE, multimodality, and Artificial General Intelligence (AGI) are impacting generative AI models, offering detailed analysis and future directions for each of these three key areas. This study does not aim to perpetuate conjecture about the unrevealed Q-Star initiative, but rather to critically appraise the potential for obsolescence or insignificance in extant research themes, whilst concurrently delving into burgeoning prospects within the rapidly transforming LLM panorama. This inquiry is reminiscent of the obsolete nature of encryption-centric or file-entropybased ransomware detection methodologies, which have been eclipsed by the transition of ransomware collectives towards data theft strategies utilizing varied attack vectors, relegating contemporary studies on crypto-ransomware to the status of latecomers [18], [19]. Advances in AI are anticipated to not only enhance capabilities in language analysis and knowledge synthesis but also to pioneer in areas like Mixture of Experts (MoE) [20], [21], [22], [23], [24], [25], multimodality [26], [27], [28], [29], [30], and Artificial General Intelligence (AGI) [31], [32], [10], [11], and has already heralded the obsolescence of conventional, statistics-driven natural language processing techniques in many domains [8]. Nonetheless, the perennial imperative for AI to align with human ethics and values persists as a fundamental tenet [33], [34], [35], and the conjectural Q-Star initiative offers an unprecedented opportunity to instigate discourse on how such advancements might reconfigure the LLM research topography. Within this milieu, insights from Dr. Jim Fan (senior research scientist & lead of AI agents at NVIDIA) on Q*, particularly concerning the amalgamation of learning and search algorithms, furnish an invaluable perspective on the prospective technical construct and proficiencies of such an undertaking[4]. Our research methodology involved a structured literature search using key terms like ‘Large Language Models’ and ‘Generative AI’. We utilized filters across several academic databases such as IEEE Xplore, Scopus, ACM Digital Library, ScienceDirect, Web of Science, and ProQuest Central, tailored to identify relevant articles published in the timeframe from 2017 (the release of the “Transformer” model) to 2023 (the writing time of this manuscript). This paper aspires to dissect the technical ramifications of Gemini and Q*, probing how they (and similar technologies whose emergence is now inevitable) may transfigure research trajectories and disclose new vistas in the domain of AI. In doing so, we have pinpointed three nascent research domains—MoE, multimodality, and AGI—that stand to reshape the generative AI research landscape profoundly. This investigation adopts a survey-style approach, systematically mapping out a research roadmap that synthesizes and analyzes the current and emergent trends in generative AI.
The major contributions of this study is as follows:
Detailed examination of the evolving landscape in generative AI, emphasizing the advancements and innovations in technologies like Gemini and Q*, and their wide-ranging implications within the AI domain.
2) Analysis of the transformative effect of advanced generative AI systems on academic research, exploring how these developments are altering research methodologies, setting new trends, and potentially leading to the obsolescence of traditional approaches.
3) Thorough assessment of the ethical, societal, and technical challenges arising from the integration of generative AI in academia, underscoring the crucial need for aligning these technologies with ethical norms, ensuring data privacy, and developing comprehensive governance frameworks.
The rest of this paper is organized as follows: Section II explores the historical development of Generative AI. Section III presents a taxonomy of current Generative AI research. Section IV explores the Mixture of Experts (MoE) model architecture, its innovative features, and its impact on transformer-based language models. Section V discusses the speculated capabilities of the Q* project. Section VI discusses the projected capabilities of AGI. Section VII examines the impact of recent advancements on the Generative AI research taxonomy. Section VIII identifies emerging research priorities in Generative AI. Section X discusses the academic challenges of the rapid surge of preprints in AI. The paper concludes in Section XI, summarizing the overall effects of these developments in generative AI.
[3] The legend entries correspond to the keywords used in the search query, which is constructed as: “(AI OR artificial OR (machine learning) OR (neural network) OR computer OR software) AND ([specific keyword])”.
![Figure 1: Number of search results on Google Scholar with different keywords by year [3]](http://hackernoon.imgix.net/images/fWZa4tUiBGemnqQfBGgCPf9594N2-mq83zz0.png?auto=format&fit=max&w=3840)
