Test & optimize | Scott Shou

[Scott] (0:00 - 0:22)

We've created a optimization calculator that's designed to lay out for you what is a complete test. And our platform builds a complete site with multiple pages, multiple ads across it, right? So it's not an A-B test because that's way too simplistic, right?

You have to think about the combinatorics, the possibilities of different things happening. And of course, there are all these different audiences.

[Dillon] (0:31 - 0:42)

What's up, lifers, and welcome to the Daily Standup with Lifetime Value, where we're giving you fresh new customer success ideas every single day. I got my man JP with us. JP, can you say hi?

[JP] (0:42 - 0:43)

I like pie.

[Dillon] (0:43 - 0:46)

Me too. What's your favorite kind?

[JP] (0:46 - 0:48)

Strawberry, cherry, pistachio, crumble pie.

[Dillon] (0:49 - 0:53)

Wow, that's intense. Mine's a Dutch apple.

[JP] (0:54 - 0:55)

What makes it Dutch?

[Dillon] (0:57 - 1:18)

And we have Rob with us. Rob, can you say hi, please? I'm more of a cake guy myself.

What's up? Oh, well, my favorite type of food is food. So there you go.

And we have Scott with us. Scott, can you say hi, please? Hey, how's it going?

Very good. And I'm your host. My name is Dillon Young.

Scott, thank you so much for being here. Can you please introduce yourself?

[Scott] (1:18 - 1:35)

Yeah, my pleasure. I'm Scott. I'm the CEO of BoardWalk Marketing.

I've been in algorithmic marketing in one form or another for about 15 years. I had a lot of fun building companies using all the different knobs and switches online. And yeah, it's been a really interesting journey.

[Dillon] (1:36 - 2:02)

All right. You said a bunch of stuff that I got questions about. I don't know if we've got time.

You said algorithmic, knobs and switches. I'm intrigued, but we have a job to do here, Scott. And that is, you know what we do.

We ask every single guest one simple question, and that is what is on your mind? We usually say customer success. I think you might want to expand that a little bit, but I'll leave it up to you.

What is on your mind, Scott? Can you tell us?

[Scott] (2:02 - 3:41)

Yeah, sure. So I'm working with a certain kind of customer, and they are the people who provide marketing traffic and users to other people, right? So normally a business thinks they need marketing in order to get more audience, get more sales.

And these are the guys who provide that stuff. Now, these guys want to expand into building their own businesses, which is great, right? They want to diversify.

They want to go deeper and build businesses with depth and longevity instead of just sending emails or buying ads on different ad networks. But then they have to overcome certain challenges themselves, right? Because if you're used to sending traffic for other people, you don't actually do the full stack of business, right?

You might look at another person's business and go, yeah, I can build that. It just looks like a website with a couple of pages, right? I got that.

But you may not see the whole elephant, right? You may not see all the things that they do for their own separate marketing or branding. They may not see all the business deals in the backend, right?

That make that business profitable and possible. So the challenge I have is these marketers are used to running one campaign at a time, usually a test at the beginning, right? And the tests are usually pretty small because all they have to do is hit some kind of a target.

And then the target's usually set by a mature business. And then they're just like, cool. I send a thousand emails.

I hit my target. I'm good, right? So when they want to build their own business and they don't have the depth, they don't have the vision of the full business.

They don't have the connections and networks. The challenges I've run into is that those guys want to run the same test because that's what they're used to, right? And then they find that that kind of falls flat.

So that's what's on my mind.

[Dillon] (3:42 - 3:49)

Rob, Scott is a personal friend of yours. So I want to make sure you get an opportunity to ask the questions you want to ask of Scott.

[Rob] (3:50 - 5:09)

For sure. For sure. Well, so Scott, we've sort of outlined the problem statement, right?

I think what's cool about what you're saying is you're talking about a reframe of what might be a decade or two's worth of someone's sort of calcified thinking about how testing works. And particularly when it comes to web traffic conversion, I think is elusive. It's mysterious.

A lot of times in customer success, we end up talking about more like enterprise relationships where you're dealing with like a person on the other side of the screen or a team on the other side of the screen. And, but we are thinking about the same fundamentals, which is conversion to goal, conversion rates to goal. So I guess I'm just kind of curious.

I know Scott, you and I have talked about this a good amount about how to try to set expectations with your clients about like about how to think differently about testing their, you know, their site conversion. And basically like how do you educate your customers about running a solid comprehensive test across, you know, a proper scientific methodology that's not just running the same test over and over again with the same simple sample size, a small sample size. So I don't know.

I'm just kind of curious, Scott, in terms of like your latest and greatest in terms of how you're thinking about solving this problem.

[Scott] (5:10 - 6:23)

Yeah. I mean, we don't know is the honest answer, but we're trying a bunch of different things, right? So we've created a optimization calculator that's designed to lay out for you.

What is a complete test? And our stuff builds, our platform builds a complete site with multiple pages, multiple ads across it, right? So it's not an A-B test because that's way too simplistic, right?

You have to think about the combinatorics, the possibilities of different things happening. And of course there are all these different audiences, right? So we're trying to give them a way to kind of estimate what the approximate damage is going to be, right?

So that's one. The others, we're just trying to discuss this early on with the customer as early as possible, setting their expectations that, hey, building a business like that has all these other steps, right? We try to mention the other parts of the elephant as it were, right?

So they can get a sense of what's involved to create some buy-in, right? When they do get to the step where they see this larger test expectation that we're trying to set. And then aside from that, we're not really sure.

We're just trying to wing it a little bit with different customers and see what works, trying different types of appeals, different types of discussions. But in general, it's communication, right? Is what we're trying as our solution.

[Rob] (6:23 - 6:30)

Do you have a practical example to share? Like if we just make up an example, I feel like that would be really helpful. We could even workshop it live here.

[Scott] (6:31 - 7:48)

Yeah. So I've got a client who sends millions of emails a day, right? So they have lots and lots of traffic.

They can point that audience gun as it were, right? At anything they want and try to turn it into a business. But when they normally do a test, it's to Google ads.

And Google ads within $50 of budget in a couple of days, with a very small amount of traffic, 100 users a day, you can figure out what Google is going to pay you, right? And so they're like, cool, that's all I need to do to do the same test, right? So build me a 10 page site with all these variations.

And here's your AI automating the testing of 10 different ads in a three ad slot on that thing and rotate it through and give me the results. And here's 100 users. So that's a real example.

Now, his goal is to turn that into an auto insurance juggernaut, right? He would love to do that. And that's great.

Auto insurance is a wonderfully valuable business, but there's a lot of deep business relationships. You need to understand what the user quality is, what the likelihood of purchase is, right? Typically, if you're having built an auto insurance lead gen before, you need about 200 leads to even get the first measurement of quality, right?

And 200 leads is not 100 users. It's actually a lot more than 100 users typically. So that's the setup.

[Rob] (7:48 - 8:35)

Yeah, I think what you're talking about is really like, it's a scientific methodology, essentially. It's like, how do you make sure that when you're running tests and evaluating conversion, that you're using an adequate sample size, a diversified sample size, and you have enough conditions and control, like rather conditions of the experiment with a clear control and clear experimental groups that you can figure out what's working and what's not. But yeah, I think it's a lot of us, we sell ourselves short when we like, for anybody listening to this show, who's like, trying to make conclusions about which playbooks are working and which aren't, for example, in customer success.

One thing to know is that you may have to try many, many, many times before reaching an actual conclusion about what's working. Strong signal.

[Dillon] (8:36 - 8:57)

Hey, Scott, I'm not too proud to say, I don't understand what the hell you're talking about, man. I get the basis of it and I understand, but man, this is so far out of my realm of understanding. So JP, why don't you go?

I think JP might be in the same boat. How do you feel JP?

[JP] (8:58 - 9:11)

Oh, no, no, no. I know all about marketing, but why talk about that? I want to know like, how did you, Scott, how did you meet Rob?

Like, how did you form this relationship? You know, what is it that really brought you two together?

[Dillon] (9:11 - 9:14)

He tries to bring it back to the relationship. You hear that?

[Scott] (9:14 - 9:24)

We were at Boston Startup Week together and started chatting there over a beer. And then I was given a card from Rob that promised me another beer. And then, you know, that's right.

[JP] (9:26 - 9:32)

So in essence, Rob is the real marketing master in this scenario.

[Scott] (9:33 - 9:37)

For one-on-one marketing, I'll kick his ass every single time. Absolutely.

[Dillon] (9:38 - 9:51)

Yeah. I think this, if I'm not being too simplistic, this is just a very sophisticated measure of statistics, right? Yes.

[Scott] (9:51 - 9:59)

And then getting an internal culture, right, at a company or at a business to adopt that proper approach. Yeah.

-: 10:48

I think we struggle with, particularly, I mean, a lot of our backgrounds are really small companies where I have not been educated on a lot of the terms you used, Scott. I think we struggle with that, with that rigor and with the feedback loop that you are espousing. We have a lot of trouble with that.

And I guess the one thing I would say to the audience is like, hey, man, I get it if you don't understand what Scott's talking about, but we should all endeavor to understand it better because it is the way a company or a process improves. It is like rock solid proven to be the way to get better. It's not throwing crap against the wall and hoping it sticks.

It's this sort of rigorous process. Is it fair to say that?

[Scott] (: 10:48

Yeah, absolutely. And so we've actually built a platform to run that rigorous process, but you still have to put in enough material into it to build the house, as it were, whereas we get like two twigs.

[Dillon] (: 11:00

Yeah. And I think maybe to talk in a language that I think a lot more customer success and SaaS folks in general understand these days is giving your AI or your custom GPT enough information to return accurate information back to you. You've got to give it context and you've got to give it a certain corpus of data to be able to not hallucinate when it responds to you.

And God forbid you make decisions or take action based on too little data that you overly trust. And so I imagine you also give some sort of confidence score to your customers around how strong the signal is coming back from those tests, right? Yeah, exactly.

I love that. Hey, look, I brought it back. I brought it back.

Scott, you're obviously much smarter than me. That's why you're hanging out with Rob, but I'll bring it back. I'll bring it back.

Thank you so much for bringing this to our attention. I'll make sure we share your website in the show notes so people know how to find you and they can ask silly questions of you as well. But for now, Scott, we do have to say goodbye.

[Scott] (: 12:08

Thank you, gents. Appreciate it.

[Dillon] (: 12:09

Thanks, Scott.

[VO] (: 12:15

You've been listening to The Daily Standup by Lifetime Value. Please note that the views expressed in these conversations are attributed only to those individuals on this recording and do not necessarily reflect the views and opinions of their respective employers. For all general inquiries, please reach out via email to hello at lifetime value media.com.

To learn more about advertising on The Daily Standup and the Lifetime Value Media Network, please reach out via email to advertising at lifetime value media.com. Find us on YouTube at Lifetime Value and find us on the social at Lifetime Value Media. Until next time.