A New Era of Tech Experimentation Is Coming

Author: Vijay Krishnan

Title: Data Science And Analytics Leader
About: Vijay Krishnan is a data science and analytics leader and a renowned expert in online experimentation, with years of experience in data science and product analytics working and consulting for multiple technology giants in Silicon Valley. Vijay holds a bachelor’s degree in engineering from IIT Dhanbad, India, and continues to operate on the leading edge of data science.
Email: krishvijay@gmail.com

Artificial intelligence (AI) is rapidly changing consumer and enterprise behavior, including disrupting classic experimentation. One of the most significant impacts is on software engineering, especially for the creation of experiments and variations in web and native apps. The advent of artificial general intelligence (AGI) will bring advancements to statistical inferences in experimentation. When artificial superintelligence (ASI) eventually takes the stage, AI will be able to seamlessly combine the science of experimentation with the creativity and intuition of the human element.

The State of AI

AI has already transformed how technology is created and used. One common application is in improving search functionality. Instead of relying on keyword matching, today’s AI-driven search engines can provide more contextually relevant results. AI is also quietly revolutionizing software development. Developers can use AI to aid with vulnerability detection, automate testing, and perform code reviews and workflow optimization. AI coding agents are also increasingly common. A GitHub article likens these agents to an assistant that can “generate code, debug existing code, and rework code to improve performance,” improving developers’ efficiency.

Traditionally, asset creation required a creative team, extensive effort, and several iterations of mockups, but AI-driven tools can now generate multiple design variations in a second. Whether for marketing, general website or app creation, or general creative assets like images, AI instantly provides users with a broad selection of design options.

AI And Experimentation

AI will evolve in stages, beginning as the technology moves closer to AGI and ending with ASI. The most immediate and likely application of AI for experimentation will happen in generating variations. Traditionally, creating variations of websites, mobile applications, or even small element variations requires extensive engineering work. AI coding agents will generate variations more quickly, making it possible to test or experiment on variations even as new features are being developed.

AI coding agents rapidly change how coding happens, and their role in software development will only grow. Earlier this year, Sam Altman, co-founder and CEO of OpenAI, predicted that 2025 could be the year that “the first AI agents ‘join the workforce.'” Various agents exist to support software development, but Altman anticipates something more like “virtual co-workers.” Although human supervision and direction will be necessary, these agents will enable an explosion of possibility, increasing efficiency in A/B testing and iterative development. AI-powered coding agents will accelerate the evolution of digital platforms and allow organizations to test ideas with unprecedented speed.

The second stage will occur as AI advances closer to AGI, altering the science and statistics of experimentation. Some of that evolution has already begun; an article published in Quanta Magazinev discusses how “a whole new universe of statistics” is already being developed to better understand AI and machine learning. As AI technology approaches true AGI, the field will continue to transform.

AI will not only improve experiment efficiency, but it may also increase creativity. Just as AI can generate hundreds of images to choose from today, it will provide experimenters with an array of possible options and variables. Organizations can test these variations simultaneously, identifying what resonates best with users.

Eventually, as AI progresses toward superintelligence, it may even be capable of bringing a more “human” touch. Experimentation is an art and a science. What is learned from past experiments enables humans to discern themes and explain their reasoning. With enormous amounts of available information and superhuman pattern recognition abilities, ASI may be able to identify themes that are harder for humans to identify and explain how it arrived at its conclusion. This fusion of scientific rigor and AI-driven creativity will redefine how organizations innovate and optimize.

AGI

While it may have seemed unlikely a handful of years ago, the expert consensus is that AGI is coming. Ilya Sutskever, co-founder and former chief scientist of OpenAI and one of the foremost researchers in the field, sees it as just a matter of time. The transformative power of AGI will be immense, especially when applied to experimentation.

AGI can potentially redefine statistical inference, such as when used for small populations, rare outcomes, measurements with high variance, heterogeneous treatment effects, and reducing Type-1 and Type-2 errors. Underpowered experiments, where there are insufficient observations to make statistically significant inferences, have been a challenge, especially for small to medium-sized companies that lack access to large datasets. Limited data sets have hindered AI, but AGI will leverage neural networks to detect hidden patterns, improve variance reduction, and refine stratification techniques. This will enhance the power of small-sample experiments, reduce inconclusive results, and help organizations make more informed decisions despite data scarcity.

Variance Reduction And Segmentation

Variance reduction is a technique used to increase the sensitivity of experimental results. A powerful method for reducing variance is leveraging stratification in randomization and measurement. Data is segmented into categories such as age, gender, and geographic location, which helps reduce noise and enables researchers to compare relevant categories.

Humans think of stratification in terms of known knowns. For an e-commerce website, the known knowns might be a customer’s purchase history or time spent on a page. Instead of relying on apparent factors like purchase history or demographics, AGI could detect behavioral or contextual signals to create multi-dimensional categories and identify previously unrecognized patterns in data.

Preparing For The Future

In experimentation, AI is a tool to support human decision-making, albeit a powerful one. Organizations that want to leverage AI require talented engineers, statisticians, and product teams to maximize this tool’s potential. It’s vital for companies to consider the risk of increasing revenue or time on site to the detriment of customers, so human oversight will remain crucial for ensuring ethical use, mitigating biases, and maintaining accountability.

As AI becomes increasingly ubiquitous, it’s critical for organizations to scale appropriately. Organizations will need to generate assets and variations to test and run their experiments, which requires computational power and efficient infrastructure. In a rapidly evolving AI landscape, it’s imperative for models and infrastructure to be designed with flexibility and efficiency in mind. AI already assists with infrastructure management to ensure that computational resources are utilized efficiently, reducing costs and improving scalability.

While the speed and volume of AI-driven experiments may seem like a mark of success, the true benchmark will be reproducibility. As the volume of experiments increases, so do the chances of random results, which makes reproducibility critical. Trustworthy experiments must produce consistent results when tested under similar conditions.

AI’s power and potential are evolving at speeds that would likely have been unimaginable even five years ago, and it’s unlikely to slow down anytime soon. AI’s predictive and pattern-recognition abilities make applying this technology to experimentation and A/B testing a logical next step. The time for waiting and watching is gone. It is critical for organizations to prepare to ride AI’s wave of innovation and transformation because it will only get bigger from here.