A/B Test The Most Probable, Impact-Maximizing, Evidence-Based And Boring Ideas, First.

Minimize risk. Maximize impact. Fractional A/B test design for hire.

Boring-First, Historical Experiment Results

Relative median effects from past experiments you can expect on projects
for metrics such as: LEADS SALES SIGNUPS REVENUE

(Medians only show the mid point. Some projects are higher. Others are lower.)

+3.8%

on smaller experiments

1 IN 3 WIN RATE

Maximize causality by isolating variables, making insights easier to reuse in the long run.

Source: GoodUI data; N = ~500

+13.5%

on larger “leap” experiments

4 IN 5 WIN RATE

Maximize gains with multiple positive compounding changes where possible (easier to detect; shorter test durations).

Source: Linowski “Leap” projects; N = 50

Let’s Discuss Your Optimization Project’s ROI

I’m available for fractional A/B test design projects that make business and stats sense. 100% response rate.

SCREEN TYPES WE HAVE THE MOST EXPERIENCE DESIGNING & OPTIMIZING FOR

Landing & Home Pages Product Pages Listing Pages Signup Funnels Checkout Funnels Pricing Pages Deliverables in Figma

HI, I’M JAKUB

Jakub

OVER

15yrs

experience designing front-end UI A/B tests

COACHED

~100

experimentation & growth teams

PUBLISHED

~500

experiments on GoodUI

HIRED BY

Microsoft
LinkedIn
Booking.com
And others

WITH

2

design degrees

AND A

Love

for stats & numbers

FLEXIBLE WAYS OF INTEGRATING WITH YOUR TEAM

As simple as:

Your Dev
Jakub

Or more complex teams:

Your
Devs
Your
PM
Your Designers
You
Jakub

JAKUB CAN LEAD:

A/B Test Design

A/B Test Analysis

Ideation & UI Design

Prioritization & Estimation

3 FRACTIONAL WAYS
TO KICK THINGS OFF

Turn My
A/B Test
Around

Discuss The ROI On This

one time fee of 3,900USD

Project StartPROJECTS STARTS WITH

You choosing an a/b test you’d like us to iterate on and turn around.

1 A/B TEST DESIGN

PROJECTS FINISHES WHEN

You re-launch the A/B test we design and we analyze the results.

INCLUDED:

Test Design & Concepting

We’ll iterate on and design 1 test for you to build (Figma).

Test Setup Guidance

We’ll work with your dev team to make sure the test is built and setup to the highest quality.

Experiment Review

We’ll check the quality of your existing experiment (SRMs, effect estimation, triggers & metrics)

Test Analysis

We’ll analyze the results and make recommendations on next steps.

Shield Icon

ITERATION GUARANTEE

If the experiment does not lead to an implementation decision, we will provide 1 extra test iteration and retest, free of charge.

MINIMUM REQUIREMENTS FOR SUCCESS:

You have your own dev team

Monthly visitors of 50k+

You have an A/B test to discuss

Maximize Gains With A Big LEAP A/B Test Design

Discuss The ROI On This

starting at a
one-time fee of 13,900USD

Project StartPROJECTS STARTS WITH

You choosing which screen(s) or flows you’d like us to optimize.

1 BIG LEAP A/B TEST DESIGN

PROJECTS FINISHES WHEN

You launch the A/B test we design and we analyze the results.

INCLUDED:

Test Design & Concepting

We’ll design 1 large leap a/b test for you to build (Figma).

Test Setup Guidance

We’ll work with your dev team to make sure the test is built and setup to the highest quality.

Opportunity Review

We’ll review the screen(s) or flows for multiple optimization opportunities.

Test Analysis

We’ll analyze the results and make recommendations on next steps.

Weekly Calls

To review test design work, capture new ideas & analyze completed tests.

Shield Icon

ITERATION GUARANTEE

If the experiment does not lead to an implementation decision, we will provide 1 extra test iteration and retest, free of charge.

MINIMUM REQUIREMENTS FOR SUCCESS:

You have your own dev team

Monthly visitors of 50k+

Iterative A/B Test
Design Marathon
(4 Month+)

Discuss The ROI On This

starting at a
monthly fee of 9,900USD

Start IconPROJECTS STARTS WITH

You choosing which metrics and screens you’d like us to optimize.

3 A/B TESTS / MONTH

PROJECTS FINISHES

After the agreed experimentation term is over (4 month minimum).

INCLUDED:

Test Design & Concepting

We’ll design 3 tests each month for you to build (Figma).

Test Setup Guidance

We’ll work with your dev team to make sure the test is built and setup to the highest quality.

Roadmap Updates

We’ll work with your PM to make sure the tests are prioritized and added to the roadmap.

Test Analysis

We’ll analyze the results and make recommendations on next steps.

Weekly Calls

To review test design work, capture new ideas & analyze completed tests.

Iterations

ITERATIONS INCLUDED

Iterations are included, given the structure of the engagement.

MINIMUM REQUIREMENTS FOR SUCCESS:

You have your own dev team

Monthly visitors of 100k+

You are already A/B testing

WHERE DO THE GOOD
IDEAS COME FROM?

1

Our Clients

Most of our ideas (good, bad and insignificant) come from over 15 years of experience designing at least ~1,000 front-end UI A/B tests. I’ve worked with selected companies such as:

Our Client Logos
2

Coaching & Learning From Growth Teams

We also talk experimentation with a variety of growth teams. I coached, learned from and shared data with ~100 teams. We share what works, what doesn’t and replicate the good stuff.

MicrosoftReverbFluke MetroBaremetricsUmbracoYummlyVivaRealVoldersMettler ToledoBackstageThomasnetDrip AgencySnocksKenhubExpert InstituteExamine.comYummly3DHubsUmbracoBomgarDigital MarketerLovehoneyJared Kayoutlet.comNorman RecordsFluke 686.comShmoodySvsound.comiBood.comFINN ElevateGetNinjasDesignlabPhorestUpliftsHomer LearningChaos GroupASICS ...
Learning From Growth Teams

Teams I coached and learned from include: Microsoft, Reverb, Fluke, Metro, Baremetrics, Umbraco, Yummly, VivaReal, Volders, Mettler Toledo, Backstage, Thomasnet, Drip Agency, Snocks, Kenhub, Expert Institute, Examine.com, Yummly, 3DHubs, Bomgar, Digital Marketer, Lovehoney, Jared, etc.

3

Actively Observing The Leaders

I also pay active attention to what leading ecommerce companies are A/B testing and how they are evolving. This is a highly impactful source of often successful optimization ideas.

Observing The Leaders

Over the years we published 100+ so called “leaked” experiments based on how leading web sites have been evolving — a really great source of ideas worth replicating.

EXPECT THE HIGHEST
QUALITY A/B TESTS

Designed
in FigmaFigma
1

CLEAR TEST DESIGN CRITERIA

Our experiments are designed with attention to detail which include setting triggers, custom metrics, owners and starting / stopping rules.

Designed
in FigmaFigma
2

ULTRA CLEAR DIFFERENCES

All our experiments contain clear differences between the control and variation with annotated screenshots. When you look back at your experiment 1 year later the differences remain clear.

3

NOTE TAKING

When we discuss tests during calls, we capture notes and decisions — it’s iteration fuel.

4

SAMPLE ESTIMATES

Our experiments come with upfront sample estimates to minimize false positive rates.

5

EXPLORATION INCLUDED

The first idea is rarely polished, free from confounders. During early exploration of any A/B test we push boundaries and generate more than one idea — widening the design space; opening conversations about new variables. Only then do we decide what’s in; what’s out or a future test idea.

Let’s Discuss Your Optimization Project’s ROI

I’m available for fractional A/B test design projects that make business and stats sense. 100% response rate.

INGREDIENTS FOR SUCCESS

1. Upfront ROI & Testing Sensitivity Check
Before we start, we’ll check the return on investment based on historical data and optimization potential. Depending on your testing sensitivity (the smaller the effect, the more sample or visitors are needed to satisfy statistical requirements) we’ll also determine what type of experiments, if and where we could run them.
2. Clear Meaningful Business Metrics With Targets
Before we start, we’ll agree to business metrics that are meaningful to you (ex: less so: clicks, page visits, time on site, etc. More of: acquisition, referrals, average order value, transactions, leads, etc). You’ll decide this along with targets needed for success. If targets are unrealistic, we’ll tell you.
3. All Optimizations Are A/B Tested
All optimization ideas that we propose will be designed as A/B tests. Controlled experimentation is one of the most reliable methods for learning about the effects of particular changes.
4. Compound If Possible
When we see opportunities to combine multiple changes together into a larger leap experiment, we’ll let you know. This way, bigger leap experiments open up the potential for larger effects (which are also easier to detect, faster, with less traffic - one of many possible strategies).
5. 80% Statistical Power With Sequential Stops
Before we run each test, we’ll do a sample size calculation based on probable effects from similar experiments ran in the past. Generally this hovers around ~3.8%, but will be adjusted lower or higher depending on the test design. We will also use sequential statistics to determine if we can stop the test earlier.
6. ROOM FOR Iteration
Iteration is an ingredient present in all of our projects as it increases the odds of success (including an increase in our ability to turn 1st failed attempt around).
7. Your Own Development Team
You will provide a dedicated front-end developer with at least 2 full days of availability per week for any project.
8. Variation Over Compromise
When we find ourselves in situations with a difference in opinion (normal), we’ll typically steer you away from compromise towards handling this through an experiment design path. This is done by adding new variations or new turning them into new test ideas. Compromise and design by committee is a recipe for unintended confounding - blurs the signal towards 0% and insignificance.
9. Multiple Sources Of Ideas
Although compromise is to be avoided, democratization of the experimentation process is highly encouraged and to be strived for.
10. Track & Correct Our Estimates
Based on all projects (anonymized) we will update our project estimates so we can better estimate in the future - independent of outcome.
Let’s Discuss Your Optimization Project’s ROI

I’m available for fractional A/B test design projects that make business and stats sense. 100% response rate.