Scaling Data Science and Experimentation: Lessons from Shuling Ding’s Journey at Tubi

Scaling A/B Testing to Drive Business Decisions

A/B testing is a critical tool for data-driven decision-making, but scaling experimentation in a streaming business like Tubi presents unique challenges. Unlike SaaS or e-commerce platforms, where controlled experiments can run over long periods, streaming companies must analyze massive datasets in real-time to optimize recommendations, ad placements, and content decisions.

Key components of an effective A/B testing framework:

⦁ Randomization & event tracking – Ensures experiment methodology is statistically sound and unbiased.

⦁ Business-friendly UI for experiment analysis – Allows GTM teams to access and interpret results without needing SQL or engineering support.

⦁ Automated data pipelines – Uses dbt to transform experiment data efficiently, reducing the need for manual intervention.

⦁ Scalable storage and processing – Data infrastructure to handle complex queries, machine learning workflows, and real-time analysis.  This isn’t unique to A/B testing, but a scalable A/B testing environment requires a well thought of foundational data model that has building blocks that can be reused and that allows separate and additional data models development.  There is always activity in the transformation layer (in this case with dbt) continuously implementing improvements as required.

Without a scalable experimentation infrastructure, teams rely on outdated spreadsheets and manual tracking, leading to misinterpretations and slow decision-making. Implementing an automated framework accelerates insights and improves confidence in A/B test results.

Accelerating Growth with Data-Driven Experimentation

A well-designed experimentation platform benefits multiple teams across the organization:

💡 For Data Teams:

⦁ Standardized processes reduce time spent cleaning and structuring experiment data.

⦁ Improved reliability of tracking ensures accurate analysis and decision-making.

💡 For GTM & RevOps Teams:

⦁ Faster insights enable better campaign optimization and content strategy.

⦁ A user-friendly interface makes it easy to access results, reducing reliance on data teams.

🚀 Business impact of scalable experimentation:

⦁ Faster iteration on product and marketing strategies, leading to better business outcomes.

⦁ Increased confidence in data-driven decision-making across teams.

⦁ Reduced bottlenecks in running and analyzing experiments, allowing teams to scale their efforts efficiently.

By investing in scalable A/B testing infrastructure and aligning data teams effectively, companies can turn experimentation into a true growth driver.

Final Thoughts: The Ongoing Balance of Technology and Business Needs

Scaling data science and experimentation capabilities is not a one-time effort—it’s a continuous process of balancing technological resources with evolving business needs. As companies grow, their data infrastructure, engineering capacity, and analytical frameworks must adapt to new challenges and opportunities.

A rigid, one-size-fits-all approach can lead to inefficiencies, whether through over-investment in complex systems too early or underinvestment that slows down decision-making. The key is building adaptable systems—ones that scale with demand, remain cost-effective, and enable teams to make better decisions without unnecessary friction.

This adaptability depends not only on technology but also on the people driving it. A strong data team needs the right mix of technical expertise, problem-solving skills, and a willingness to learn new technologies as the landscape evolves. For managers, the challenge is to balance individual capabilities and interests with the demands of an ever-changing data environment, ensuring that both the team and the business continue to thrive.

These insights come from Shuling’s experience at Tubi, where she navigated the complexities of scaling data infrastructure, aligning teams, and implementing a robust experimentation framework. Her experience and insights highlights the importance of adaptability, collaboration, and strategic investment in technology to keep pace with rapid growth.