tl;dr

Statistically modeled conversions in Facebook don’t have revenue tied to them, therefore ROAS is being under-reported. This implies we as advertisers need to figure out other ways to measure accurate performance, like using AOV from Google Analytics and purchases from your desired attribution window to build a more accurate ROAS.

What the heck happened to my Facebook ROAS?

OK, so maybe “lying” is a bit of a stretch, but it got you to click and I AM a direct-response marketer after all :)That’s right. There is revenue missing from your purchases being attributed to Facebook. See this for example:

You can see 2 purchases were attributed to this ad. But what you don’t see is any purchase conversion value (revenue) or ROAS. Initially I chalked this up to a glitch, but when these “glitches” kept happening I knew it was worth investigating deeper.And in case you were wondering, this isn’t because we don’t have those parameters flowing through to Facebook through the pixel and Conversions API, we do.So then what the heck is going on…?

Here’s a little context on iOS14.5 (if you need it)

Because of Apple’s iOS14.5 update on April 26th, Facebook rolled out many changes to its reporting, conversion tracking, and targeting. If you’re reading this article, I assume you’re plenty familiar with the fiasco, so I won’t bore you with further context than that. If you’re just returning from La La Land and have no clue what’s going on with iOS 14.5, here’s some additional context.‘Social Media Gurus’ have been claiming that they’re seeing massively inconsistent and zig-zagged results on Facebook since the rollout, and while that’s certainly true to a point, we at Elumynt haven’t seen the pandemonium that seems to be plaguing the DTC Twitter chats.When iOS 14.5 went live on April 26th, there wasn’t some mythical “ROAS destroyer” that dropped its hammer crushing everyone’s ROAS overnight...  One of the reasons why ROAS remained relatively stable (with more of a gradual decrease) is likely because not everyone updated to iOS14.5 at once.What did happen overnight though, was the way Facebook reported and attributed conversions. The most obvious being the removal of the ability to see conversion events by attribution window in-platform. Facebook hadn’t made this part clear, and the ads community took to Twitter to ameliorate the pain.

• Via @andrewfoxwell

After some discovery and comparing platform reporting (like, actually looking at the Facebook Ads UI) to API reporting (pulling data into our data warehouse), we figured out that what the platform was showing was whatever attribution window you’re optimizing to. If you’re optimizing for two different attribution windows in one campaign, you won’t see any conversion metrics at the campaign level.For example, let’s imagine there are two ad sets in a campaign, one optimized to 1 Day Click, and one optimized to 7 Day Click. When you’re looking at the Campaign level, you’ll see no conversions for that campaign.

When you drill down to the ad set level, you’ll see the conversions for each ad set, but they won’t be aggregated in the summary row (because you’re not comparing apples to apples).

When you drill down into a single ad set on the ad level, you’ll see how many conversions were attributed to each ad. These conversions and other metrics like ROAS and revenue will be aggregated, since every ad you’re drilled down to is optimizing to the same attribution window.Following that logic, if you were to not select an ad set and then move on to the ad level, you wouldn’t see these conversions aggregated.

These changes basically made it impossible to use the Facebook Ads UI to understand what’s performing well. This is why it’s so important to know how to maneuver the platform metrics these days, and to make sure your clients understand that what they’re looking at in-platform may not mean what they think it means. Instead of using the UI to aggregate data or evaluate attribution windows, you must pull data from the Facebook API. We use both Supermetrics and our own data warehouse to gather the datapoints we need and customize the attribution for each business.  Without data from the Facebook API you’ll have to drill down like this on every campaign every time you do a pulse check.

Now, here’s where it gets… weird

Back to Facebook “lying” to us about revenue… again, taking a bit of creative license here. Facebook isn’t lying (at least not on purpose), but the data in the Facebook Ads UI certainly is misleading.One of the changes Facebook preached loud and clear(ish) about is the deployment of statistical modeling for both click-through and view-through conversions when users have opted out of tracking, which makes perfect sense.

What they did not make clear, and completely failed to mention (to the best of my knowledge and research), is what I’ve come to realize more recently —  a statistically modeled conversion wouldn’t have revenue attributed to it.Above, we showed you an ad that has two purchases and no revenue, and we’ve established that these are likely statistically modeled conversions. But what about ads with more purchases? How could we tell if there are conversions without attributed revenue on an ad with lots of conversions?

Let’s take a look at another example. On this ad you see 6 purchases attributed, and $60.97 in revenue, which would leave us with an AOV of $10.16 (assuming we aren’t double tracking conversions, which has been ruled out). However, this client doesn’t even sell products under $14.99, and their AOV from Facebook purchases according to Google Analytics Paths Report is $34.88, so we are assuming that out of the 6 purchases on this ad, 2, maybe 3 of them have revenue attached. This would imply that on this ad, 50-66% of revenue isn’t being attributed, therefore ROAS is being significantly under-reported in platform.

But, does it matter?

You may be thinking, okay, I get it, but do these statistically modeled conversions carry any weight?We think yes, with a grain of salt.

...Sorry, I couldn’t resist.Anyway, Apple is saying, hey Facebook, we know that you know a ton about your users and activity online, but we need you to not tie these conversions to a specific person.Now, keep in mind, we can’t guarantee this is what’s happening, but this is what we feel COULD be happening (and the data seems to agree with us)... Facebook said, okay fine. For people who don’t want us to track their data, we’re going to start using other signals and activity, current conversion rates on PDPs, and some statistical magic and modeling to determine if they converted, but we won’t say who it is (eliminated conversion data in breakdowns) and we won’t put a conversion value on it, so businesses can’t find an order in their CRM for that value on that day and tie it to a specific Facebook ad they converted on.

Here’s an example of the data and how ROAS is currently calculated in platform, not mind-blowing math.

That said, there’s one more adjustment we need to take into account before we have a worthy formula.We saw that Google Analytics AOV was consistently higher than the AOV attributed in Facebook prior to April, so must account for that discrepancy in our Adjusted ROAS calculation.Here we’ve plotted AOV from Facebook vs AOV according to Google Analytics paths report. Through April, the discrepancy between these numbers stays quite consistent, however when May hits, Facebook’s AOV starts to drop, and in June much more dramatically.

Next we plotted the percent difference between the two to show this more clearly. We’re accounting for this in our Adjusted ROAS by taking a conservative average of the AOV discrepancy from the months leading up to iOS14, (-15%). Since we see that Google Analytics AOV is typically higher than what we see in Facebook, using the full AOV would be overcorrecting and inaccurate. Note as of mid-June, this difference has nearly doubled, and it doesn’t look to be stopping yet.

This makes our new formula:  (GA AOV * FB Purchases * AOV Discrepancy Adjustment) / Spend = Adjusted ROAS. Here we can see it in action.

Here’s a way to confirm the logic and figure out just how drastically this could be affecting your accounts. When we look at attributed revenue and conversions between Google Analytics Paths report (1-day lookback window) vs 1 Day Click attribution in platform, we see 4.8% more purchases in Google Analytics (FB attributed, 1 day lookback). If Facebook is attributing revenue to each purchase, we should see a similar difference in revenue. However we actually see a much more drastic discrepancy; there was 60.54% more revenue attributed in Google Analytics than to Facebook’s platform.If we use our Adjusted Revenue formula to see what our attributed revenue would be using Google Analytics AOV and purchases from Facebook’s platform (608 Purchases * $33.9 AOV * 1+ (-.15) AOV Discrepancy Adjustment = $17,519 ), that gap shrinks to 18.9%, and that revenue is much closer to what Google Analytics attributed on a 1 Day lookback.

So, what should I do now?

Not only does this make Facebook Ads reporting inaccurate, it affects the business as a whole if  we were to make decisions based on the in-platform ROAS; we might scale down spend, or turn off ads and/or audiences that don’t seem like they’re performing. We might spend more resources than needed testing new creative concepts, and we might stress out a lot more than we need to.From an algorithmic optimization perspective, there aren’t huge implications if you’re optimizing for the event rather than value. However, if you’re using value optimization (based on revenue, in this case) for an ad like you’ve seen here that has two purchases but no revenue tied to it, Facebook isn’t being given any positive reinforcement for those conversions (not ideal).

Let’s recap:

• There are conversions taking place, and being counted in Facebook that have no revenue attributed
• The reported ROAS is therefore lower than what the ads are likely contributing, as evidenced by Google Analytics paths report

Our recommendations are iterative as we continue to dig into these changes and what they mean, but what we are currently suggesting as a starting point is performing a similar analysis to what we’ve done.These are a few routes that we’re vetting for each of our clients as next steps:

1. Track a CPA goal rather than solely a ROAS goal when reporting on channel specific performance.
2. Correlate ROAS and Adjusted ROAS alongside MER (Media Efficiency Ratio) and ACoS (Ad Cost of Sale)
3. Continue tracking directional changes among platform performance in Google Analytics, as tracking here should remain constant as it is based on UTMs
4. When reporting on ROAS, compare the platform ROAS against the Adjusted ROAS Use the AOV from Google Analytics Paths report to calculate revenue: (GA AOV * FB Purchases * (AOV Discrepancy Adjustment+1)) / Spend = Adjusted ROAS

And if you haven’t already, make sure you’ve implemented Conversions API. Also note that when you set this up through Shopify, most of your Auto Advanced Matching fields will be turned off, so head back into your pixel settings to turn them back on.These solutions may not work for every client, and the landscape is continuing to change, so periodically reevaluate this for your accounts to determine if it’s still proving to be accurate.