Back to Case Studies
SberDevices2020-2022900-person startup

Enterprise Test Automation Architecture for Smart Devices

Building comprehensive test automation from scratch for a major banking innovation department developing smart devices to compete with Amazon Alexa.

95%
Time reduction
1,670+
Test scenarios automated
90 min
Regression cycle time
14
Team members built

The Situation

I joined a major Russian bank's new innovation department focused on developing smart devices and voice assistants, essentially a 900-person startup operating within a regulated banking environment. The department was tasked with creating a competitive product to rival Amazon Alexa, targeting both consumer markets and specialized hotel solutions.

When I arrived, testing was entirely fragmented—developers performed ad-hoc testing based on individual preferences, with no standardized approach or reusable assets. Manual QA teams were overwhelmed by growing regression needs, with each incremental change requiring significant validation effort.

The initiative was barely a year old with no production release yet, but faced aggressive go-to-market timelines and high executive visibility as a flagship innovation project for the bank.

The Challenge

I faced multilayered challenges requiring both technical and organizational solutions:

  • Regulatory constraints: Enterprise banking environment where every access request required weeks of compliance reviews
  • Greenfield build: No existing test automation team or infrastructure to build upon
  • Technology diversity: Web applications, Android devices, voice assistants with NLP, complex backend integrations
  • Cultural resistance: Development teams accustomed to their own testing approaches
  • Aggressive timelines: Pressure to capture market share before competitors

The solution needed to cover disparate technologies while building trust with development teams and establishing a new centralized automation function from scratch.

Technical Architecture

I developed a comprehensive multi-framework architecture with purpose-built solutions for each testing domain:

Web UI Framework

Java-based framework with optimized Selenium implementation for web application testing

API Integration

Dedicated API/integration framework with configurable mock server using JSON schema

Android Framework

Kotlin-based Android UI E2E testing framework aligned with native app development

Parallel Execution

Scalable execution architectures supporting 40 concurrent test threads for maximum throughput

Implementation Strategy

Team Structure

Built and led a 14-person automation team structured to balance specialization with cross-training—each engineer owned a specific domain area (web, Android, API integration, NLP) while contributing to core framework development.

Phased Rollout

Implemented strategic rollout approach, working with one development team at a time, starting with web applications as the foundation to build trust and demonstrate value.

Support-First Positioning

Positioned the automation initiative as a support function rather than oversight mechanism, emphasizing our role in "taking headaches away" from development teams.

Risk-Based Prioritization

Prioritized automation efforts based on business risk and release schedule requirements, focusing on high-value regression scenarios first.

Transformational Results

The framework we built transformed testing capabilities across the entire smart devices division, delivering measurable business impact:

Coverage Achievements

  • • 100% automation of voice assistant testing (1,670+ scenarios)
  • • Complete web-based testing automation (1,200+ scenarios)
  • • 56% API/integration testing coverage
  • • 80% Android UI testing automation (190 scenarios)

Efficiency Gains

  • • 95% reduction in regression testing time (2 days → 90 minutes)
  • • 40 concurrent test threads execution
  • • Monthly stakeholder demonstrations enabled
  • • Critical release blockers consistently caught pre-release

The automation framework became fundamental to maintaining release cadence and enabling monthly stakeholder demonstrations. Executives were amazed by areas where automation was complete while pushing for acceleration in areas still in progress.

Most importantly, the framework consistently caught critical issues before release—from broken voice assistant skills to mid-flow Android UI failures and backward compatibility breaks—proving its value in a startup environment where rapid changes often introduced unexpected defects across the technology stack.

Need Enterprise Test Automation?

I specialize in building scalable test automation architectures from the ground up for complex technology stacks.