Validating a Faster Path for Policy Testing

The Brief

The organization needed to shorten the time required to test and launch policy changes without weakening the controls required for safe validation. Testing access, technical knowledge, and execution were concentrated in tenant-team application developers, creating repeated handoffs and long turnaround times across policy teams.

About This Case Study

This case study examines how an assignment to improve policy testing uncovered a broader operating-model problem. By correcting the research direction and mapping the policy lifecycle from end to end, I identified how overlapping role terminology had concealed the concentration of access, technical knowledge, and testing responsibilities within one application developer role. That discovery shifted the product strategy from improving one developer's workflow to distributing appropriate testing capabilities across the tenant team.

Starting From the Wrong End of the Problem

Original Direction

Reframed Direction

Original DirectionImprove how the tenant-team application developer manages testing

→

Reframed DirectionReduce the concentration of testing work in one role

Original DirectionPreserve the existing access and role structure

→

Reframed DirectionDistribute appropriate access and capabilities across the tenant team

Original DirectionOptimize individual testing tasks

→

Reframed DirectionReduce repeated handoffs between technical and non-technical users

Original DirectionBegin with the tenant-team application developer's workflow

→

Reframed DirectionBegin with the end-to-end policy lifecycle

Original DirectionAddress the visible workload

→

Reframed DirectionAddress the access, knowledge, and operating-model conditions creating it

Project Timeline & Scope

This year-long assignment, one of two I worked during the engagement, took place from late 2024 into 2025 within a newly formed data and AI organization responsible for modernizing policy management and testing. Multiple product teams built different parts of the platform, while tenant teams used it to perform policy work. Within those teams, access and technical knowledge varied, and one application developer often supported the entire team.

My scope was to understand how policy work connected across the platform and its tenant teams, document the end-to-end lifecycle, and translate that understanding into a clearer product direction. The assignment concluded before enterprise rollout, but after early testing produced measurable improvements in policy turnaround.

Why This Problem Could Not Be Ignored

Policy testing depended heavily on the application developer, who was often the only tenant-team member with both access to the testing tools and the technical knowledge required to use them. Policy owners and other non-technical tenant-team members relied on the application developer to retrieve data, run and adjust tests, interpret results, and support validation before launch.

As requests, results, and adjustments moved between policy owners and the application developer, work accumulated around one technical role. The delays affected policies governing fraud flags, credit offers, promotional eligibility, and other customer-facing decisions. Testing had to become faster without weakening the controls required for safe validation.

How the Problem Was Initially Framed

Leadership wanted to reduce the bottleneck slowing policy testing and delivery. An internal hackathon had produced an early testing concept, and the initial research plan called for interviewing tenant-team application developers to understand how they performed the work and determine how the concept could be developed into a working product.

However, the first interview list directed me to platform developers, the teams responsible for building the policy platform, rather than the tenant-team application developers using it to build and test policies. Both groups were referred to as "developers," but they performed different work and interacted with the platform for different reasons. That terminology had obscured which developers the research needed to reach.

When "Developer" Meant Two Different Things

The first interviews revealed that platform developers could explain how the technology was being constructed, but not how tenant teams built, tested, validated, and launched policies. Product managers and stakeholders had assumed that the developers building the platform were the same developers building policies within it. The shared title had concealed the difference in their work and relationship to the platform.

Without a shared understanding of who performed the work, I could not determine where the testing bottleneck began or what the product needed to support. I redirected discovery toward mapping the policy lifecycle from end to end, identifying the people involved, the work each role performed, and the dependencies connecting them.

Mapping the Policy Lifecycle End to End

I expanded discovery to identify the roles involved in creating, testing, validating, approving, and launching policies, along with the tools, access requirements, and dependencies that connect each stage. The lifecycle map showed that policy owners understood the intended changes but often lacked the access or technical knowledge required to test them. The application developer had become the bridge between policy decisions and technical execution.

Each test required the policy owner to explain the intended change, wait for the application developer to retrieve data and run the test, review the results, and then return with additional questions or adjustments. That cycle could repeat several times before the policy was ready for validation and launch.

The delay was not contained within one testing task. It was produced by repeated handoffs across the policy workflow. This shifted the product question from "How might we help the application developer complete testing work more efficiently?" to "Which testing activities could other tenant-team members perform directly with the appropriate access, guidance, and safeguards?"

Where Policy Testing Became Bottlenecked

1. Define the policy change

2. Test and revise

3. Validate and prepare for launch

Policy owner

1. Define the policy change

• Defines the intended change

2. Test and revise

• Reviews results
• Requests adjustments

3. Validate and prepare for launch

• Confirms the intended outcome

⇅Repeats with each revision

Application developer

1. Define the policy change

• Receives the request

2. Test and revise

• Retrieves data
• Configures and runs tests
• Resolves technical issues
• Returns results

3. Validate and prepare for launch

• Supports validation before launch

Platform developers built and supported the shared platform but sat outside this loop; they were not the users performing policy work.

Policy knowledge and technical execution were separated across roles, causing each revision to generate another handoff.

The Bottleneck Was Built Into the Operating Model

Policy owners held the business and policy knowledge needed to decide what should change and evaluate whether the results matched the intended outcome. Application developers held the system access and technical knowledge needed to retrieve data, configure tests, and execute them. Neither role could move the work forward independently.

This division created a structural bottleneck. Even minor questions or adjustments required another exchange, and requests accumulated around an application developer who often supported the entire tenant team.

The opportunity was not to remove the application developer from the process or transfer every technical responsibility to policy owners. It was to determine where guided access, clearer results, and built-in safeguards could allow other tenant-team members to complete appropriate testing activities independently while preserving technical oversight where it was still needed.

Redefining Who the Product Needed to Serve

The original direction centered on the application developer because that role performed the technical work. Lifecycle research revealed that the product also needed to support the policy owners and other tenant-team members whose decisions and revisions drove it.

I reframed the product around different levels of participation rather than giving every user the same tools or responsibilities. Policy owners needed guided ways to evaluate changes and understand results. Application developers still needed the technical capabilities required to configure complex tests, resolve exceptions, and provide oversight.

Designing around the tenant team could reduce requests concentrated in one role while allowing policy decisions and technical validation to move together more efficiently.

Translating the Operating Model Into Product Requirements

Guided

Policy owner capabilities

• Initiate appropriate tests
• Select historical or custom data through guidance
• Review policy conditions
• Interpret result summaries

Shared

Visibility and safeguards

• Show what was tested
• Show changed inputs and conditions
• Make results understandable
• Signal when technical support is required

Retained

Application developer capabilities

• Configure complex scenarios
• Resolve exceptions
• Perform advanced troubleshooting
• Handle work requiring deeper access or expertise

I translated the role differences into product requirements. Policy owners needed guided ways to initiate appropriate tests, adjust policy inputs, and determine whether results matched the intended business outcome. Application developers needed technical controls for complex scenarios, exceptions, and work that could not be completed safely through a guided experience.

The experience also needed to show what had been tested, which inputs or conditions had changed, what the results indicated, and when technical support or additional validation was required.

These requirements established a role-appropriate direction: routine testing activities could move closer to policy owners, while complex work remained with application developers. This reduced unnecessary handoffs without removing the safeguards, access controls, or technical oversight required for validation.

Designing a Guided Testing Experience

I developed guided workflows and prototypes that allowed policy owners to initiate tests, select historical or custom data, review policy conditions, and interpret result summaries without requiring the application developer to mediate every step.

The experience made technical complexity more manageable through guided inputs and clearer results while keeping complex configurations, exceptions, and technical decisions within the application developer's responsibility. It also connected test setup and evaluation to the organization's existing development processes rather than treating testing as an isolated activity.

Validating the Direction Before Scaling It

I evaluated the workflows and prototypes with intended users before enterprise rollout. Validation examined whether users could set up tests, work with appropriate data, understand the output, and determine whether a policy behaved as intended. It also tested whether appropriate activities could move beyond the application developer without sacrificing necessary guidance, controls, or technical oversight.

Early testing supported that direction. In one existing-policy cycle, turnaround decreased from approximately six months to two weeks. In one new-policy cycle, it decreased from approximately eighteen months to six months. These results were limited to early cycles, but they demonstrated that reducing repeated handoffs within the tenant team could materially shorten policy delivery.

The evidence gave the team a validated direction for continued development while preserving application developer involvement for complex configurations, exceptions, and work requiring deeper expertise.

Turning Early Evidence Into an Aligned Product Roadmap

The early results supported continued development but did not represent a finished enterprise solution. I used the lifecycle map, prototype, and validation findings to distinguish what the guided experience already supported from what still needed to be developed before it could scale.

The roadmap included expanding testing workflows, improving how users selected and worked with data, and making results easier for non-technical users to evaluate. It separated activities that could be safely guided from those requiring deeper technical expertise, additional controls, or continued discovery.

These artifacts gave product and technical stakeholders a shared view of the roles, bottlenecks, and product boundaries shaping policy work. They aligned the team around a broader strategy: distribute appropriate testing capabilities across tenant teams while preserving technical oversight where the complexity or risk required it.

What I Changed Beyond the Interface

My contribution extended beyond designing a testing experience. I identified that the research had been directed toward the wrong developer group, redirected discovery toward the end-to-end policy lifecycle, and exposed how access, knowledge, and responsibility were concentrated within the operating model.

I translated that diagnosis into a product strategy supporting different levels of participation across the tenant team. By connecting lifecycle research, requirements, experience design, validation, and roadmap priorities, I helped the team move from an incomplete understanding of the bottleneck to an evidence-backed direction for reducing policy turnaround safely.

What the Work Demonstrated

Early results showed that the bottleneck was not inevitable. One existing-policy cycle decreased from approximately six months to two weeks, and one new-policy cycle decreased from approximately eighteen months to six months. Because these were two early cycles before enterprise rollout, they validated the direction but did not establish organization-wide impact.

The work demonstrated the value of investigating the operating model before defining the solution. What appeared to be one developer's workflow problem was rooted in how access, knowledge, and responsibilities had been distributed across the tenant team. Correcting the research direction and mapping the full lifecycle gave the team an evidence-backed path toward faster policy delivery without weakening the safeguards required for safe validation.