>In the end I decided the only real way to tell if what we were doing was working, was to turn it all off, for several months to see if there was a measurable difference.
What?
If this person genuinely thinks that there is no way to measure an ad's performance while it is running, they are either too ignorant to be in business, or they hired someone who lied to them.
If you’re full e-commerce the only metric that really matters is conversions. How do you know your ads aren’t just taking credit for customers who were going to buy anyway?
You can get great CTR and conversion rate running ads against your brand name. But if your SEO is sound you’re pretty much just canibalizing your organic traffic.
I’m not aware of a way to determine whether an ad just claimed credit for a customer it didn’t earn. I’m interested to learn though.
> How do you know your ads aren’t just taking credit for customers who were going to buy anyway?
The word for this in the business is "incrementality", and there are several ways of measuring it. The simplest conceptually, is that you run two ads to two random groups of users: one for the product and one for something irrelevant like a charity. Then you compare conversions between the two groups.
(There are fancier ways to do it that don't require you to spend half your budget on an irrelevant ad, but that's the basic idea.)
The article suggests that they are comparing between two placements: the organic search placement and the paid placement (ad). I don't see any explanation why that is invalid. Indeed the literal meaning of incrementality would suggest a comparison of f(A) and f(A,B). Not quite "sampling with replacement", but if that's heresy to you you're welcome to quit reading now.
The protocol for doing so would involve studying what happens when the ad is present versus when it is not. The goal is conversion, scrupulously defined as people who click one or the other and subsequently purchase. Total conversion could go up, down, or stay the same. The only way the ad "wins" is total conversion increases, and even then maybe. If organic conversions went up when the ad was present you'd have a research problem! (The effect could be time-based, i.e. "awareness", or it could be a confounding externality.)
The protocol you suggest would seem to be removing the organic placement when the ad is present, that is: one or the other. On its face this sounds more "researchy" to me. Putting feasibility aside, it would plausibly be attractive to an advertising professional, but I would espect the customer to ask "why?" and I don't see the answer to that question. What's the motivation for this approach? I can see that it makes the advertising professional's contribution crystal clear, but why should the customer pay for it?
But hey I don't have 30 years of advertising experience, nor do I consider myself a statistician or machine learning expert. I do however have over 30 years of experience as an internet plumber and (more importantly here) data sous chef, so I've tasted a lot of ingredients in a lotta stews and have a solid grasp of experiment design and causality.
You work for the customer: consider that some avuncular advice.
The problem with the approach in the article is that there are lots of reasons conversions can vary over time that are unrelated to what you're trying to study. If you have to use time to distinguish treatments, your best option is to alternate time periods (ex: one day on, one day off). But you can almost always run your two groups simultaneously, giving them different treatments, which allows you to eliminate the effect of timing noise.
That is what you're talking about, how slicing up your signal before transmission impacts your ability to receive it. Here's a Jupyter Notebook which will maybe make your head explode... I mean if you like math.
Unless you control the ad platform, you can't make a 'no ads' control group; if you want to track the user, you've got to show them some ad; so an unrelated PSA is an ok option.
Time doesn't work very well. There are lots of reasons why you will see different behavior at different times, so it really only works if you are trying to check something with a very large effect size.
Much better is to run your control group and experiment group in parallel, like I described above.
Is your objection that showing ads for something irrelevant could affect whether the user converts?
So the experimental setup is: you create an ad for your product, you create an ad for your charity, then you compare the populations of people who click on your ad for your product with the population of people who click on your ad for the charity and see if there is any difference in conversion rates between the two populations.
How do you ensure that the people who see/click on your product ad aren't already a population more likely to convert to your product than the population of people who see/click on your charity ad? Sure, you can target the same groups of people - but that only goes so far, the ML algorithms backing the ad selection process will still preferentially show your charity ad to people likely to click on charity ads and show your product ad to people likely to click on your product ad.
I am unfamiliar with how these experiments work on the advertiser side. Incrementality is easy to measure on the platform side if advertisers report conversion metrics to you.
ML messing with you is an important consideration when you are trying to run this sort of experiment on a platform that you don't understand very well. The most simplest reliable way to run this experiment is to give the ad network an HTML creative that looks like:
<script>
var treatment = readTreatmentFromCookie();
if (!treatment) {
treatment = Math.random() < 0.5 ? CONTROL : EXPERIMENT;
writeTreatmentToCookie(treatment);
}
if (treatment === EXPERIMENT) {
showAd(PRODUCT);
} else {
showAd(CHARITY);
}
</script>
The creative is opaque to the network, which means it's not going to be able to do anything fancy like you're describing.
Many networks offer the ability to run a fully supported incrementality study, where they effectively use one company's ads as a control for another's, but this is a version of an experiment that you can run even if you don't trust the ad network at all.
Attribution is not my area of expertise. But could you not track them to see if they convert? This is possible both on the advertiser side and the platform side, provided the advertiser reports conversions to the platform.
If you chose "not to show ads" to the control group, all the slots of the control group where you would have shown ads will have ads you don't control, including your competitors' ads.
If you show a neutral ad, you know exactly how much seeing your brand in ad drives revenue compared to seeing an irrelevant ad.
Hey Jefftk, this is a human dimension I wasn't acknowledging elsewhere. Theory is one thing and money is another. ("whiskey is for drinking, and water is for fighting over" or something like that -- Samuel Clemens)
So in the real world I bet this is what you're talking about: "but there are clickmonkeys out there" or words to that effect.
[edit:] But to be honest, I'm not sure it directly answers the question about entanglements between ad and no ad.
The goal in showing an ad for something irrelevant is to have a control group: one where it is very unlikely that your interaction is going to affect their purchasing your product.
I agree that only conversions matter, but you should be able to track whether a customer originated from an ad click vs. SEO, whether they clicked one ad and had 4 sessions since, etc.
While it's not a perfect system, you should know with a reasonable degree of accuracy how many dollars you get back for every dollar you put in to ad spend.
As far as the SEO traffic issue, this can also be accounted for once you have scaled your ads beyond the levels of what SEO could yield. At low budgets this can be more of an issue, at high budgets much less so.
I don’t know how counting sessions or source of click matters. You still don’t know whether the customer would have ultimately found you without the ad.
Put another way, the only way to tell if a customer will still find you without the ad is to not run the ad.
If you have sufficient traffic you might be able to measure this by turning your ad spend down rather than off and measure the bottom line impact. But other than gross spend changes there’s no way of telling for sure that the ads do anything.
This is like the paid search results that come up when you search for a specific company name. The company's site is just below those paid results, but often the same site is in the paid results but they are further up so are more likely to be clicked.
> You still don’t know whether the customer would have ultimately found you without the ad.
Sure, maybe 100 customers would eventually find you. But if you buy the traffic, you can make them all find you on the same day.
> But other than gross spend changes there’s no way of telling for sure that the ads do anything.
I'll use the most fundamental example:
If someone clicks your ad and buys the product during that session, you know the ad worked. If you keep increasing your ad budget every day, and you are consistently returning $2 for every $1 you spend, you know it's working. Turn off the ads, and the revenue goes away. You'd be surprised how many people make their living doing this.
>If someone clicks your ad and buys the product during that session, you know the ad worked
This is not as clear as you make it seem. Let's say there's a hypothetical product that a consumer only buys once every 5 years. If someone clicks your ad and then immediately buys the product... what if they would have bought it anyway, tomorrow, or 5 minutes from now, without the ad? How can you test that counterfactual?
If I buy cat litter online once a month every month for 5 years by going to example.org/catlitter -- but then they decide to start advertising on facebook, so now I click the facebook ad once a month to buy cat litter from the same site, are the ads "working"?
The idea is because it's happening at scale. If you're getting hundreds or thousands of customers per day, the likelihood of this happening gets lower by some statistical proportion, especially if you don't have the organic exposure.
Also, definition of working: You make more than you spend, and if you stop the ads, you stop making as much.
That said, I'm not arguing for 100% accuracy either. It's certainly not. Simply that it's possible to be more accurate than not, which leads to profitability.
One more also- a comment above about how I failed to mention I'm not talking about paid search ads so much as other types like FB, YouTube, banners, etc.
> I agree that only conversions matter, but you should be able to track whether a customer originated from an ad click vs. SEO, whether they clicked one ad and had 4 sessions since, etc.
An ad-click could have been an organic click if the ad wasn't there though, which is where the complexity is.
The ad might have great conversion, but if the customer would have clicked an organic link that navigated to your site anyway then the ad is taking credit for an organic sale.
While you're not necessarily wrong, the distinction lies in how many people see you organically.
If I get 100 organic impressions/day, and then spend $N to get 500/day, I'm speeding the process up, and nearly guaranteeing to get in front of people who would never see me organically.
*Note: I'm largely not talking about paid search ads. Those are definitely an area where you can end up competing with yourself. Sorry to anyone who I replied to earlier, and wasn't clear enough on this with.
> you should be able to track whether a customer originated from an ad click vs. SEO
That's easy to track, but answers the wrong question. It answers whether the person clicked on the ad or the organic link. What we actually want to know is whether this person would convert even without seeing an ad. The only way to answer whether a person would still convert without seeing an ad is to make sure that they don't see an ad.
My major failure in this thread was not specifying from that start that I'm not so much talking about paid search ads (google ads), as much as I mean FB, YouTube, etc. That's the arena where you've really gotta question if you're competing with yourself.
On the flip side of that, most of HN seems to think purely in terms of paid search ads, when there are many other types of online advertising that exist, and that don't have this issue baked into them.
Such is the nature of internet dialogue, I guess lol.
Are you conflating a call to action on a channel you already control with advertising? Because in that case you might try f(A), f(A,B) and also f(B): could be a synergistic effect, or they could just click on anything.
> whether they clicked one ad and had 4 sessions since, etc.
That's not trivial to do, both from a technical point of view (browsers - rightfully - fight these kinds of tracking attempts) but also legal (GDPR mandates that the customer opts into this kind of tracking but they have no incentive to do so).
You're right, this is actually a lot harder to recently, with the iOS updates that occurred a year or so ago. It's a space that evolves very quickly.
But this is a good place to point out that, yes - there is nuance to all of this - and I didn't really mean to turn it into a thesis on online advertising (lol) so much as to say:
It's still easy to be more accurate than "Turn it off and see what happens".
Even a semi-sophisticated media buyer is going in with a plan, and some method of measurement.
GDPR doesn't stop me from having ad1.example.com and ad2.example.com landing pages. Aspirationally perhaps, but technically no. I think this is different from fonts.gstatic.com in that it doesn't need to follow people around the internet and it's also not info necessarily going to a third party.
> How do you know your ads aren’t just taking credit for customers who were going to buy anyway?
If you are doing advertising right you build a funnel from awareness through to conversion and track every part of it so you then a/b test your adverts and channels.
Getting advert channel fit is the key to success and can only be done by testing and measuring your adverts. The fact that op couldn't measure the effectiveness of their adverts shows they weren't doing this and their bad outcome should be 100% expected.
If you are just buying adverts in the hope you get more conversions then you are putting those adverts out to die. Burn your money you will probably get more eyes for doing that than untested adverts.
> How do you know your ads aren’t just taking credit for customers who were going to buy anyway?
If you can't answer that question, then you're not running a proper advertising campaign. You're just throwing money around blindly. Plenty of people do have good answers to that question, though. Unfortunately, plenty of people don't.
Well, for our business, proper advertising campaigns have opened new channels for growth, allowing us to reach new people, and build our email list significantly, with a predictable ROI. It's pretty easy to tell if an ad is working when you turn it on, and your email list growth rate instantly doubles -- and all the traffic is coming from a new traffic source. (Obviously, if we were to run Google ads on search traffic, we would check whether our organic search traffic dropped when we turned the ads on, but that's a different point.)
> It's pretty easy to tell if an ad is working when you turn it on, and your email list growth rate instantly doubles -- and all the traffic is coming from a new traffic source.
Maybe for a real tight definition of instantly, but generally this is not true. Imagine the following scenario:
1. You run an ad for ComapnyName.
2. An enthusiastic customer promotes your business in a local bar.
3. The bar attendees search for CompanyName and click on the first result.
Because you ran an ad for CompanyName, the first result is going to be your ad. You will see very nice ROI on that ad. What you won't see in any stats, is that these people would have probably found the organic link anyway, because they were already motivated and searching for CompanyName in particular.
> Obviously, if we were to run Google ads on search traffic, we would check whether our organic search traffic dropped when we turned the ads on, but that's a different point
That wouldn't help either in this imaginary scenario, because these bar attendees are a spike in traffic. Plus if the ad has been running for a longer period, then you won't have accurate organic search traffic stats anymore either, because it's already cannibalized.
"Imagination" is the core of the problem here. Plenty of people are imagining various scenarios, whereas people with successful advertising campaigns are just raking in the money -- no need for imagination.
I've turned off plenty of ads when they stopped working. I didn't need to turn them off to discover they stopped working.
Sure, you can imagine a dozen scenarios where an ad campaign doesn't work. Fortunately, I've managed to learn how to focus on the reality of the situation -- and have reaped the rewards in the process.
> you won't have accurate organic search traffic stats anymore either, because it's already cannibalized.
That' not true for us -- as I said before, a successful campaign for us brings in new sources of traffic from new marketing channels. One of the early lessons I learned was to not cannibalize what's already working. For example, we grew our Facebook Page (and email list) from scratch, when we had no Facebook traffic, by using Facebook advertising.
Operating under different brands -- even just for testing purposes, is a one another of the way we deal with this. Of course you can continue to imagine scenarios where we might be making mistakes. I do that as well -- it's called planning. Though, none of that really matters until money is spent (and made or lost).
First let me say that ads in general definitely work, I'm definitely not arguing against that.
What I'm talking about in specific is search ads where the ad is for a term that ranks organically high anyway.
It's not just imaginary either, I've done a lot of over-the-shoulder customer observing. Just recently I saw a friend search for "dropbox" and then click on the first result in Google, which is a paid ad by dropbox for dropbox. They rank #1 anyway!
Now in dropbox's case it might be worth it, because they have enough competitors who would like to steal that ad spot. However for most businesses that's not the case for their top terms.
One problem is the ad industry is quite good at finding ways to take credit for conversions that would have happened regardless, and analytics are all systematically biased in a way that conveniently maximizes the ability to do so.
The idea to simply turn off ads and see the effect is born of a healthy distrust of ad analytics industry bluster.
> One problem is the ad industry is quite good at finding ways to take credit for conversions that would have happened regardless
This is a great counter-point when someone mentions "use conversion tracking!" Conversion tracking is great, but not if they store a cookie for 7/15/30 days and "award" the conversion to the ad, when the customer took a different and varying path to purchase. Sure the ad contributed "some" to the conversion, but not 100%.
I strongly suspect they sometimes show the ad and credit it for the conversion after the user shows initial interest in some specific product, i.e. when the user was already highly likely to buy before seeing the ad. This would explain why you're often flooded with ads for something after you search for it, go to its website, or purchase something there. (We've all had that experience of buying something and then being tailed by ads for that thing for weeks afterwards, even though no sane person would think we were going to buy that thing again in such a short time frame.)
A story to illustrate: there was once a pizza store that had two guys go out into the city to distribute promotional coupons. The coupons had codes on them so the business could attribute sales to each coupon distributor. John went out into the city and tried his best to drum up new business. Chad stood next to the door of the pizza place and handed a coupon to anyone who was walking in. After a month, 98% of the coupons used were from Chad. Chad got a big bonus and John was let go.
Companies that sell ads will do everything they can to convince you there is a measurable difference, but often while the metrics presented to the client are real, they do not always translate to real world impact.
There is no easy way to determine whether this is the case without just comparing when the ads are on and off and somehow dealing with the confounding variables, which isn't easy.
Well that works in some ways, but it can also be very hard to disentangle the actual causal impact of an ad campaign! Absent running an experiment, you don't really know if people would or would not have bought through some other channel without your campaign.
The most striking example is probably the experiment Tadelis convinced eBay to run on their brand-name ads, though the subtlety applies to less obvious cases of questionable ad spend too — https://faculty.haas.berkeley.edu/stadelis/Tadelis.pdf
I'll just reference my other replies to the first part here, but thank you for that link too. I'll have to give it a read when I have more time later today
> If this person genuinely thinks that there is no way to measure an ad's performance while it is running, they are either too ignorant to be in business, or they hired someone who lied to them.
Doing a blackout month is a legitimate technique.
I've deeply studied ad performance metrics and they are NOT as conclusive as people might think. If they were, you could just keep increasing the amount of $ spent and the conversions would go up. Ok, not exactly, there is usually a diminishing returns aspect, but you get the point.
A great example of why this requires more analysis is branded vs non-branded search terms on AdWords. Let's say you spend $10k/month on the brand "Mattel" (aka branded) and $10k/month on "toys" (aka non-branded). Mattel likely gets you like a 20x ROAS because people are probably searching for a specific Mattel toy. It's also potentially likely that if you completely turned off branded search you would net the same results.
> this person genuinely thinks that there is no way to measure an ad's performance while
This is the case in many situations. The time between initial visit to site to conversion can be weeks, months, or years. Meanwhile, customers will visit site from various platforms. There is no way of attributing value to a single click, it needs to be approximated as an aggregate.
This feels like a no true scotsman. I have doubts that the vast majority of digital advertising platforms do anything other than clutter websites, waste bandwidth, annoy people, and pay a small slice of tech employees. I'm open to seeing research on the topic (and I am pretty sure there is some), but what I've read is that most advertising has insufficient statistical power, thus confidence of advertising outcomes being anything but random flukes is low.
It's funny to me that everyone has a "take" on advertising, and yet 99% of the successful B2C brands, including multi billion dollar international ones, continue to advertise. Every platform that starts as ad-free gets pressured to allow ads and most of them acquiesce.
It's not that these companies love throwing their money away. Maybe there's just something they know that you don't?
Maybe it's that getting your product in front of the right people at the right time has immense value. And many platforms have opened up spaces for you to attempt to do that if you pay them for the space. Maybe paying for the wrong space at the wrong time is a waste of money.
Maybe determining the right place and time to get in front of people is a skill as well as an entire profession. Maybe that entire profession can't be reduced to an absolute binary of does it work or doesn't it.
I'm very curious if you work in advertising/adtech, otherwise your "take" is no different than the ones you're criticizing.
I have worked on the data side of a pretty wide range of roles across the marketing/adtech spectrum for over a decade and think their is a lot of good reason to be skeptical of the claims of the advertising world.
Tim Hwang is also an insider in this industry and wrote an entire book (The Subprime Attention Crisis) on the issues with the current state of advertising. I work in a very different area from Tim (he's legal) but I can tell you that book almost bored me with how obvious all of his complaints where.
> It's not that these companies love throwing their money away. Maybe there's just something they know that you don't?
I've seen the data that many of these companies don't. As many others have said, simply dismissing advertising as a "scam" is too extreme, however there are a lot of really big issues in the industry and extreme skepticism of the advertising industry is well warranted.
The reasoning of "if the system is fundamentally broken, then why are so many people participating in it?" is easily dismissed with any of the major financial crises we've seen. This same logic could be falsely applied to the pre-2008 financial crisis "if these ratings are so wrong then why are so many experts putting so much money in them?"
Personally I don't know anyone who works on the "how the sausage is made" side of advertising that isn't at least somewhat skeptical of the whole system.
I work in advertising and know how the sausage is made. However I don't have equity in an ad agency, I don't profit off of promoting advertising. I participate in these conversations to help people understand. Also because they often piss me off.
> The reasoning of "if the system is fundamentally broken, then why are so many people participating in it?" is easily dismissed with any of the major financial crises we've seen.
The financial crisis was about companies making money, which they love to do. Advertising ad spend is about companies spending money, which they hate to do. Unless someone can explain why everyone wants to subsidize advertising agencies and ad platforms.
> Personally I don't know anyone who works on the "how the sausage is made" side of advertising that isn't at least somewhat skeptical of the whole system.
Skeptical of what, exactly? If you use 3rd party impression and click tracking tools, attribution modeling software, and statistically significant testing, I am genuinely confused as to what there is to be skeptical of.
I think the people who say they know "how the sausage is made" and still hold skepticism of "the whole system" are maybe not as knowledgeable as they may think.
In good faith, I am definitely skeptical of a few things. Whether ad platforms are really trying to prevent spam. How 3rd party DSP audiences are built and why they think people are ok with using them having no idea how they are made. Whether or not apps and devices really are spying on people. Whether people are aware of what "privacy" means from an advertising perspective.
But I'm not skeptical about the users that come to my site or which marketing efforts are working or not working.
> The financial crisis was about companies making money, which they love to do. Advertising ad spend is about companies spending money, which they hate to do. Unless someone can explain why everyone wants to subsidize advertising agencies and ad platforms.
Advertising is certainly in the interest of the ad agencies, and the employees of companies whose job is to either manage outside advertising or develop/execute advertising in-house. It's possible there could be a company with a lean team of advertisers, doing just the type of work that makes sense. But within any organization, leaders want to have larger teams because it is seen as a marker of respect. It also allows a leader to command a higher salary.
I don't know if these forces are sufficient to have spun the entire advertising industry out of nothing. But I do know that there are significant forces looking to build up advertising both inside and outside of companies.
I have no idea whether advertising works or not, but I see this "successful businesses do it, therefore it it works" argument applied to so many different things and it always baffles me. Besides not having much substance beyond an appeal to authority, I don't think I've ever seen an example of a company that doesn't engage in some number of financially wasteful behaviors with dubious or at least unquantifiable value.
The vast majority of successful B2C brands are owned by companies like Unilever, or are Apple, etc. I don't think there's an easy comparison between the ad goals and spending of these companies and those of smaller B2Cs trying to get off the ground. Maybe your own justification is comparing Apples to oranges, pun intended, and besides the point.
> 'It's not that these companies love throwing their money away. Maybe there's just something they know that you don't?'
Those companies that have a Sign-up for our email and get 10% off your first order pop-up, you mean?
Those companies paying for clicks to 404 pages or 'this item is out of stock', etc?
Those companies that ask you if you have a discount coupon just before you enter your card details for something you are already buying?
I get your point but I wouldn't assume big companies always know what the are doing when it comes to advertising. Sometimes they employ lots of people and some of those people don't actually have a clue what they are doing.
I agree with your broader point that you shouldn't just assume large brands always know what they're doing but other than the 404 example I'm not sure these are actually indications of companies not knowing what they're doing.
> Those companies that have a Sign-up for our email and get 10% off your first order pop-up, you mean?
I get that this can be annoying but plenty of companies do A/B tests and find that it works for them. I suppose it could mean they're just following some fad and don't know what's going on, but it doesn't have to mean that. This is especially true for companies that have long sales cycles or are in categories where lots of comparison shopping is common. Getting someone into your email funnel can be more important than anything else.
> Those companies that ask you if you have a discount coupon just before you enter your card details for something you are already buying?
Where else in the funnel would you have them apply their coupon? Maybe my perspective is different because we do a lot of offline advertising, but if someone comes into the site off of a print coupon they're going to expect to be able to put it in somewhere and get their discount. If we don't put it in the order flow they're either going to not purchase or we're going to get a lot of customer service calls from people trying to redeem their coupons.
Fair comment. I think we can agree there are case where such things can be used to great effect but that there also are businesses throwing money around and hoping something sticks.
One big mistake businesses make is seeing another business doing something and assuming it must be working. Which is one of the points the OP was making.
I'm sure the executives at these big corporations would love to see stock prices go up if they could find a few extra million dollars per quarter in useless revenue negative activity that they could easily cut while having no impact on sales.
The reason I say this though, is because if they knew how to run ads properly they would have been tracking their results from the start, and would have known much earlier whether they were getting a return on their investment.
If your only way of measuring advertising results is to "turn it off", you're just flying blind, and one can't expect success with a (lack of) strategy like that.
That said, you are not wrong that a meaningful percentage of advertising is being run with similarly insufficient statistical power, and to that I would say those businesses are also incorrect, for the most part. I delineate because at some point, say when you're Apple or Microsoft, you are so big that "brand awareness" advertising takes over performance advertising. For the most part though, I'd say those aren't the types of businesses we are discussing in a context like this one.
Tracking advertising effectiveness is ridiculously difficult and multiple people inside and outside your company are incentivized to overstate impact.
Statistical power for example assumes independence which can be very difficult. Great you spend X million to convince people to buy an AC in March, did you actually benefit or would those same customers want an AC as soon as the first heat wave hit? Spreading demand can be useful, but it’s also really easy to to draw false conclusions from statistics if you don’t understand the domain.
And that’s just one of the many pitfalls involved.
How do you know any strategy is attributable to success or failure without testing it?
Pre/post analysis may be temporally correlated but this isn't proof because you haven't captured a baseline comparison.
A/B and MAB testing are helpful but not magic bullets.
Shapley values (marginal impact) is a nice mathematical outcome to have for multitouch attribution but as usually implemented is only a single statistic and can be a fluke without additional testing.
A phenomenon which reminds me of the canonical survivorship bias story. In WWII they conducted studies to determine where the bullet holes where on aircraft which returned from bombing sorties, in order to determine which parts of the aircraft required armour. It took a statistician to point out that they actual needed to armour those places where they rarely saw damage on returning aircraft, as those parts are most likely the parts where being hit caused the aircraft to not return at all.
Sometimes it requires a bit of a leap of imagination in order to resolve these things.
Causal inference has made a lot of improvements since WWII, and "if" the advertising company knows what they are doing they run effective A/B or MAB testing; that said, statistical power is typically low because of insufficient sample size for individual companies.
You could pool all ads together, but since each advertising company is independent you get into all kinds of weird path dependencies.
While I wouldn't claim to be an adtech practitioner, I did at one point help a few F500 work through conceptual models of multitouch attribution and other statistical issues. These are very nontrivial issues -- proving advertising effectiveness is very difficult!
It's a problem in that those keywords are some of the places where you are at least likely to be generating counterfactual conversions: most of that traffic was probably coming to anyway.
Appealing to effeciency is not an appeal to purity, which is what a no-true scotsman is.
A no true scotsman in this regard would be more along the lines of redefining advertising to not include any activities that OP described, i.e. OP wasn't doing true advertising. GP is not doing that here, because GP is acknowledging that OP is doing advertising, but doing it poorly.
It felt like one because the original comment puts anyone who doesn't willingly support advertising claims as not knowing how advertising works. The classification creates a false dichotomy whose classification is "only a group that does not know advertising would do X."
If you spoke to any growth marketer worth their salt at any D2C company they will have incrementality testing and split tested traffic to prove without a shadow of a doubt that advertising works. The real challenge is scaling without losing efficiency.
>It never ceases to amaze me how many people/businesses have no idea what performance advertising is.
Yikes, that's incredibly unfair and arrogant.
It's amazing to you that, because most people don't have to actually employ performance advertising, they don't know what it is? That someone whose passion is cooking, and decides to open a restaurant, might not have that advertising knowledge? That someone - in the case of the OP - whose focus is writing software that helps research an automotive vehicle's life history, might not know everything you do about advertising?
No you're right, as someone who takes these things seriously, I should hold myself to a higher standard than to paint in such broad strokes. Thanks for checking me there.
Allow me to rephrase from a more compassionate perspective:
I don't expect any of these people to devote the type of effort that I have into this knowledge.
But I do wish they knew this stuff, because with even a little bit of this knowledge, they could have the power to make their own restaurant/software shop/insert_small_business more successful than it otherwise could have been... which may even be the difference between them successfully running said business vs. having to take a job they don't like.
Ultimately, it's a be the change you wish to see situation, I suppose.
This comes across as an arrogant view point. Would you know how to take off the heads of the engine in your car and rebuild it? No? Wow! I'm amazed that you'd have no idea how to do something that isn't your direct line of work.
People running small businesses that are so wanting for ad buys to work for them don't spend years honing their performance advertising skills. They don't even spend time looking it up to know it's a thing (first time I've heard this phrase myself). They see all of the advertisng they are subjected to about why buying ads is important, and so they start where they can.
Instead of making fun of people for not knowing something that they shouldn't need to know about, why not corner the market by providing non-insulting services to get them the results they need? Or at the least, be able to point people in the direction of where to get those services?
Condescension for the sake of patting yourself on the back is just gross.
You are totally correct, thank you— I addressed this to jjulius above, since they pointed out the same thing. Sometimes one must be reminded not to be flippant on the internet, it's all too easy, and I don't want to be that person.
>Instead of making fun of people for not knowing something that they shouldn't need to know about, why not corner the market by providing non-insulting services to get them the results they need? Or at the least, be able to point people in the direction of where to get those services?
You're right. We all know the value of the person who makes complaints without offering solutions.
If anyone reads this and would like some honest help in this area, send and email to the address in the 'about' on my profile, and I'll try to point you in the right direction (It only looks sketchy because it's a forwarding address, I'm sure you understand.)
> why not corner the market by providing non-insulting services to get them the results they need?
That market doesn't exist because it requires buy in from the business, businesses that see the value in targeted and performance driven advertising do it in house because its so valuable, other businesses just don't do it because they see no value in it and they see no value in it because they don't do it.
Advertising works when its targeted but most businesses see advertising as just trying to shout as loud as possible. They take this theory and shout at every one they meet hoping this will convert them to a customer and are amazed when shouting at people has the effect of driving them away rather than pulling them in.
They come to the conclusion that advertising doesn't work not that its there technique, they shouted so loud and at every one how could anyone possibly shout louder or at more people? and when they stopped shouting sales went up! obviously advertising doesn't work.
>That market doesn't exist because it requires buy in from the business, businesses that see the value in targeted and performance driven advertising do it in house because its so valuable, other businesses just don't do it because they see no value in it and they see no value in it because they don't do it.
That goes against the entire concept of the advertising agency though. If advertising is so important, why staff it out to a 3rd party when you could do it in house? If this in house thing was the way to go, why is Maddison Ave so powerful?
Most businesses that take advertising seriously will have someone in house and then out source the specifics to specialists but the larger strategy is done in house. In the UK generalist agencies like av browne have been hurting bad the smaller agencies have been closing.
>That market doesn't exist because it requires buy in from the business, businesses that see the value in targeted and performance driven advertising do it in house because its so valuable, other businesses just don't do it because they see no value in it and they see no value in it because they don't do it.
All that sounds like to me is that the company providing OP's posited "non-insulting services" would just need to make sure they're marketing their product correctly and demonstrating value properly.
Advertising is definitely a lot more complex than simply: spend more money on it -> sales go up.
When tobacco ads were banned, tobacco companies started making more profit, because they had to spend less on advertising. Turned out their advertising was mostly to steal customers from each other, and didn't really lure in new users. So the ban actually helped them.
Short-term, yes. Long-term, I think no? Tobacco advertising typically focused on making smoking cigarettes seem cool and glamorous, and banning the ads may well have been a large component of why it no longer seems so.
> Tobacco advertising typically focused on making smoking cigarettes seem cool and glamorous, and banning the ads may well have been a large component of why it no longer seems so
That's one hypothesis. Another is that we've known for ages that tobacco smoke causes disease[0] since[1]
> Lung cancer was once a very rare disease, so rare that doctors took special notice when confronted with a case, thinking it a once-in-a-lifetime oddity. Mechanisation and mass marketing towards the end of the 19th century popularised the cigarette habit, however, causing a global lung cancer epidemic. Cigarettes were recognised as the cause of the epidemic in the 1940s and 1950s, with the confluence of studies from epidemiology, animal experiments, cellular pathology and chemical analytics. Cigarette manufacturers disputed this evidence, as part of an orchestrated conspiracy to salvage cigarette sales. Propagandising the public proved successful, judging from secret tobacco industry measurements of the impact of denialist propaganda. As late as 1960 only one-third of all US doctors believed that the case against cigarettes had been established. The cigarette is the deadliest artefact in the history of human civilisation. Cigarettes cause about 1 lung cancer death per 3 or 4 million smoked, which explains why the scale of the epidemic is so large today. Cigarettes cause about 1.5 million deaths from lung cancer per year
As someone who has run ads across a variety of mediums, online and offline, some products benefit from being in the public consciousness, others dont. Identify which products or services benefit from certain types of advertising will help enormously otherwise its just throwing good money after bad.
The OP's point 1 fails to recognise the filter bubble though, I think some SEO companies capitalise on this, but it simply works like this, if you keep googling your website, Google will eventually make it one of the top links in the result for YOU, not anyone else and for some website owners/companies, thats enough for them, and it doesnt bring in any more sales or revenue.
I would love to see solid evidence that advertising for an established product is meaningfully effective at increasing revenue beyond the cost to produce it.
Another fun one— next time a public figure dies, especially a slightly more obscure one, just google them/their death.
Whole first page is just poorly written fiverr style articles, all by people for whom English is clearly a second language, at best.
Eventually, legit stuff gets on the first page too, but theres a few instances where weeks later the first result is still one of these garbage seo-torture sites.
Has really made me lose my faith in google lately.
Since apparently it's not connected to credit score, what stops you from just not paying it?
If small claims court is genuinely their only recourse, that sounds like something I might try my odds in, especially if I have a compelling case that I've been screwed.
I get what they're getting at here, but isn't cycling still wildly more efficient than walking/running? Maybe they haven't changed because they are already reasonably optimal.
Biking is wildly more efficient than running. Heck, if you’re on the large side (250-300pounds), you burn as much energy (~0.25kWh or ~200Calories in terms of human food, which—unless you’re eating almost entirely bulk staples like corn meal—has a really high footprint, especially fruit and meat) to run a mile as it takes to drive a Model 3 for that same mile. Biking is a quarter that. (Walking is a bit more efficient than running, but not much.)
The usual study/reference for this is someone biking at ~10mph. Bicycles become progressively less efficient at higher speeds due to air resistance.
Fairings and reduced frontal area (recumbent/supine positioning) can help this considerably. The combination is commercially sold as velomobiles and they're not especially popular but they do work. As an example, dedicated amateur enthusiasts do 3 hour century rides in commercial velomobiles regularly.
FTA: It has survived, largely unchanged in dimension, philosophy and architecture, since the time of The Boer War, precisely because it is not that bad. It is actually reasonably good at low to moderate power outputs.
What makes you say "reasonably good"? It's excellent. Ten kilograms of metal makes you three times faster at a quarter the energy spend, I'd say it's amazing.
