How to Conduct Effective Creative Testing for Mobile Apps

App Store Optimization
CRO
Aug 6, 2024

“I know that half of my advertising budget is wasted; the problem is, I don't know which half” – retail magnate John Wanamaker said this about his marketing budget more than 100 years ago. However, there are now more opportunities to measure the impact of advertising on the audience and make it more effective. The RadASO team explains how to do creative testing.

What is creative testing?

Creative testing in marketing is a process that involves trying out different creative elements to determine which ones are most effective in achieving goals. It includes experimenting with various content formats, designs, messaging, and other aspects. 

Even more insights from ASO & UA pros in ASO Busters on Discord!

The main goal of testing is to find the optimal combinations to increase your audience engagement, improve conversions, and overall effectiveness of advertising campaigns. 

Here is an example for better understanding:

Let's imagine that a company has launched a new product – eco-friendly water bottles. 

Then it needs to launch an advertising campaign on social media. The main goal is to attract as many leads as possible and increase sales. To do this, the team creates several variants of ads that differ in elements, i.e:

1. Images:

  • option A – product photo on a white background;
  • option B – visual in a natural environment (for example, in the steppe, forest, or beach).

2. Title text:

  • option A – “ Contribute to the preservation of the planet with our eco-friendly bottles!”;
  • option B – “Your new favorite water bottle - stylish and eco-friendly!”.

3. Call to action (CTA):

  • option A – “Order now”;
  • option B – “Learn more”.

Then, the company launches several ad groups, where each group is a combination of different images, headlines, and CTAs:

  • group 1 – image A, headline A, CTA A;
  • group 2 – Image B, headline B, CTA B;
  • group 3 – Image A, headline B, CTA A;
  • group 4 – Image B, headline A, CTA B.

The next step is to analyze the results of each ad group by key metrics. Examples of metrics:

  • click-through rate (CTR);
  • cost per click (CPC);
  • conversions (purchases)
  • cost per conversion (CPA);
  • engagement (likes, comments, shares).

The final step is to determine the most effective variant, based on the collected data. For example, the team may find that the combination of image B, headline A, and CTA B has the highest CTR and conversions.

This example shows how creative advertising testing helps to identify the most effective elements of an advertising campaign.

Attribute
Value
Format
PNG
Color space
Display P3 (wide-gamut color), sRGB (color), or Gray Gamma 2.2 (grayscale).
See Color Management.
Layers
Flattened with no transparency
Resolution
Shape
Square with no rounded corners
Attribute
Value
Format
PNG
Color space
Display P3 (wide-gamut color), sRGB (color), or Gray Gamma 2.2 (grayscale).
See Color Management.
Layers
Flattened with no transparency
Resolution
Shape
Square with no rounded corners
Word
Frequency in service
Qimai(China)*
AppTweak
Mobileaction
Appfollow
课程 (course)
5387 (high)
56 (mid)
5 (low)
34 (mid)
外语 (foreign language)
5699 (high)
35 (mid)
5 (low)
34(mid)
播放 (play)
5496 (high)
35 (mid)
5 (low)
35 (mid)
*Search index in China provided by Qimai and based on multidimensional calculations such as search results, downloads and relatable keywords.
App Store
Google Play
1. Icon impact on user choice
Important, but not more than a video with screenshots, as they take up most of the screen
Icon is displayed with several other icons in search results, and therefore it has the greatest impact on the user's choice
2. Icon localization
1 icon for all locales
Can be localized for each country
3. Screenshots quantity, orientation
From 1 to 10, any orientation
From 2 to 8, any orientation
4. Screenshots the most important are the ones that are visible before you start scrolling
  • Vertical screenshots: 3 are visible
  • Horizontal screenshots: only the first one is visible
  • Vertical screenshots: 4 are visible
  • Horizontal screenshots: only the first one is visible
5. Screenshots size
Large — the content is legible
Much smaller — the captions are quite illegible
6. Video quantity
Up to 3
One video that must be uploaded to YouTube
7. Video orientation
Any orientation
Horizontal orientation prioritized
8. Video autoplay
Plays automatically for 30 seconds without sound
You need to click on it to play
9. Application cover
Displayed on the application and developer pages
There is no cover
10. Graphics update
After release only
Whenever

Why do you need to conduct testing?

Creative performance can improve the effectiveness of marketing campaigns and help you achieve your business goals. 

5 reasons to start creative testing:

Expense optimization

By understanding which elements of advertising bring the best results for your brand, you can optimize your marketing spend. This will allow you to reduce the expenses on less effective ads and invest more in those that bring the highest ROI.

  1. Increase campaign efficiency

If you know which creatives (images, text, CTAs, etc.) work best, you can focus resources on the most effective options and increase the overall effectiveness of your campaigns.

  1. Measuring efficiency

Specific metrics make it possible to measure the effectiveness of advertising campaign elements.

  1. Better understanding of the audience

Insights into target audience behavior are important for business. They can be used to create new ideas and personalized campaigns that will resonate with the audience.

  1. Конкурентна перевага

Being the first to identify and implement effective strategies through testing is one of the competitive edges in the market.

Thus, testing helps to better understand audience preferences. Due to effective combinations, campaigns can be optimized, which in turn helps to achieve results and increase return on investment (ROI).

Device or context
Icon size
iPhone
60x60 pt (180x180 px @3x)
60x60 pt (120x120 px @2x)
iPad Pro
83.5x83.5 pt (167x167 px @2x)
iPad, iPad mini
76x76 pt (152x152 px @2x)
App Store
1024x1024 pt (1024x1024 px @1x)
Source language
Target language
How much longer (+) or shorter (–) the text in the target language is
English
French
21.18%
English
Spanish
19.52%
English
Italian
17.91%
English
Deutsch
16.67%
English
Dutch
13.80%
English
Portuguese (Portugal)
14.29%
English
Portuguese (Brazil)
12.96%
English
Polish
9.33%
English
Russian
9.11%
English
Czech
3.70%
English
Arab
–6.25%
English
Japanese
–39.68%
English
Korean
–44.04%
English
Chinese (Simplified)
–61.97%
English
Chinese (Traditional)
–63.80%
*Search Ads Popularity (SAP)shows the popularity of the search term from 5 to 99.

Types of creative testing

A/B testing (split testing)

One of the most popular types of testing for your brand is a b testing, where two different variants (A and B) of the same element (for example, a headline, image, CTA) are tested on a part of the audience. This helps to determine which of the variants works better. 

A/B testing involves creating two variants (A and B) of the same element. Variant A is called the control variant (usually the current or old version of the creative), and variant B is called the test variant. Then these variants are shown to different groups of users simultaneously and their effectiveness is compared by certain metrics. 

Tools for A/B-testing. In Google, this is Google Ads Experiments. In ASO in App Store Product Page Optimization is used for this purpose, in Google Play - Store listing experiments.

Proper use of A/B testing will help you make an informed decision to achieve better results.

A/A/B- or A/B/B-testing

A/A/B testing is a method that compares two identical control (current) variants (A and A) with one variable variant (B). 

A/B/B testing involves comparing one control (current) variant (A) with two identical modified variants (B1 and B2).

This type of testing helps to obtain more accurate confidence, as a larger sample reduces the possibility of random errors or mistakes in the results.

Multivariate testing (MVT)

MVT is a simultaneous testing of several variants of different elements (for example, headlines, images, CTAs) to determine the most effective combination. It differs from A/B testing, which compares only two variants of the same element.

Multivariate testing involves creating several test variants for each element under test and their combinations. For example, if you are testing headlines, images, and CTAs, and there are three variants of each, the number of combinations will be 3x3x3=27.

It can include the following components:

  • headlines;
  • images/videos;
  • texts;
  • CTA buttons.

Due to this type of testing, we can speed up the optimization best practices and find effective solutions to achieve business goals, as we have the ability to test creative elements and their combinations at the same time.

Conversion Lift tool for Meta

This tool allows you to measure the increase in advertising performance on Meta platforms. When you run a test, you create two similar audiences - a test audience and a control audience. People from the test audience see ads, while those from the control audience (also called the “control sample”) do not.

The test results provide answers to the following questions:

  1. How effective is advertising on Meta platforms?
  2. Do I need to increase my campaign budget?
  3. How to increase the effectiveness of advertising on Meta platforms?
  4. How many conversions have you managed to get thanks to advertising on Meta platforms?
  5. Do the results obtained with the current attribution model match the results of a controlled experiment?

Requirements for testing by Meta:

  1. The ad account has a campaign with a start date within the last year and a budget of at least $5000. Also, the campaign must have at least 500 optimized conversions with attribution settings of “1 day after click”, “7 days after click”, or “1 day after view”. 
  2. For campaigns longer than 90 days, the requirements change proportionally. For example, a 180-day campaign requires a budget of $10000 and 1000 optimized conversions. Optimized conversions include only the ones you specify and only for the events for which you performed optimization.

You can create a test in the Experiments tool or using the Meta account representative (if you have one).

Note. To perform any testing, you need to have a sufficient amount of traffic. Then the test will be considered valid. The less traffic there is, the fewer test variants should participate in the test.
Device
Spotlight icon size
Settings icon size
Notification icon size
iPhone
40x40 pt (120x120 px @3x)
29x29 pt (87x87 px @3x)
38x38 pt (114x114 px @3x)
40x40 pt (80x80 px @2x)
29x29 pt (58x58 px @2x)
38x38 pt (76x76 px @2x)
iPad Pro, iPad, iPad mini
40x40 pt (80x80 px @2x)
29x29 pt (58x58 px @2x)
38x38 pt (76x76 px @2x)
Search term
Translation
SAP* English (U.S.)
SAP* English (U.K.)
truck games
games with trucks
62
39
lorry games
games with trucks
32
jail games
prison games
29
8
prison games
prison games
32
24
*Search Ads Popularity (SAP)shows the popularity of the search term from 5 to 99.
App Store
Google Play
What can be tested?
  • Short description
  • Long description
  • Icon
  • Feature graphics
  • Screenshots
  • Videos
  • Screenshots
  • Videos
  • Icon (has to be uploaded to the build*)
Number of simultaneously running tests
5 tests (each test is valid within a single country.

You can choose a default country test (details below): then it will run in all countries where there are no localized graphical or textual materials.
1 test (the test can be immediately extended to all countries where the application is available or opt for specific countries as needed)
The number of test variants that can be tested with the current version in the store
Compared with a maximum of 3 new variants
Can a test be launched while another item is under review?
Yes
No
Mandatory formats for screenshots uploaded to the store
6.5
  • 6.5
  • 5.5
  • 12.9 (if there is an iPad version)
*Build – is a new version of the application. Updating the icon is only possible when updating the application version in the store. In other words, the term "build" refers to a specific version or variant of the application that is ready to be downloaded and installed on the users' devices. It contains all the necessary files and data for users to install and use the application.
More about optimizing graphic elements in the App Store and Google Play can be found in the article 'Graphics in Mobile App Promotion in the App Store and Google Play (ASO) – How to Optimize Graphic Elements.'

Where to find ideas for tests

  • competitor analysis – observe what your competitors are doing and how they are doing it;
  • communication with customers – collect feedback, it will become a valuable source for your ideas;
  • internal brainstorming sessions – invite employees from different departments to discuss ideas (marketing, sales, customer support);
  • trend research – keep track of current trends and adapt them to your topic;
  • social media – Instagram, Facebook, LinkedIn, Pinterest, and TikTok are sources of inspiration;
  • creative agencies – cooperate with creative agencies;
  • training and courses – participate in training, webinars, and online courses;
  • collaborate with influencers, they often have fresh ideas and insights into the audience;
  • explore successful case studies of other companies in your industry and beyond.
Device size or platform
Screenshot size
Requirement
Screenshot source
6.5 inch (iPhone 13 Pro Max, iPhone 12 Pro Max, iPhone 11 Pro Max, iPhone 11, iPhone XS Max, iPhone XR)
1284 x 2778 pixels (portrait)2778 x 1284 pixels (landscape)1242 x 2688 pixels (portrait)2688 x 1242 pixels (landscape)
Required if app runs on iPhone
Upload 6.5-inch screenshots
5.8 inch (iPhone 13 Pro, iPhone 13, iPhone 13 mini, iPhone 12 Pro, iPhone 12, iPhone 12 mini, iPhone 11 Pro, iPhone XS, iPhone X)
1170 x 2532 pixels (portrait)2532 x 1170 pixels (landscape)1125 x 2436 pixels (portrait)2436 x 1125 pixels (landscape)1080 x 2340 (portrait)2340 x 1080 (landscape)
Required if app runs on iPhone and 6.5 inch screenshots are not provided
Default: scaled 6.5-inch screenshotsAlternative: upload 5.8-inch screenshots
5.5 inch (iPhone 8 Plus, iPhone 7 Plus, iPhone 6s Plus)
1242 x 2208 pixels (portrait)2208 x 1242 pixels (landscape)
Required if app runs on iPhone
Upload 5.5-inch screenshots
5.5 inch (iPhone 8 Plus, iPhone 7 Plus, iPhone 6s Plus)
2048 x 2732 pixels (portrait)2732 x 2048 pixels (landscape)
Required if app runs on iPad
Upload 12.9-inch iPad Pro (3rd generation) screenshots
5.5 inch (iPhone 8 Plus, iPhone 7 Plus, iPhone 6s Plus)
2048 x 2732 pixels (portrait)2732 x 2048 pixels (landscape)
Required if app runs on iPad
Upload 12.9-inch iPad Pro (2nd generation) screenshots
Price
Traffic Source (Self Reporting Networks)
Facebook
Google
X (Twitter)
Apple Search Ads
Cohort reports
Impression tracking
Audience segmentation
At extra charge
Custom Dashboards
Custom Reports
At extra charge
Advertisement Cost
DAU/MAU (Stickiness)
Raw Data Export
At extra charge
At extra charge
At extra charge
API Reporting
Search term
Translation
SAP* French
SAP* French (Canada)
soldes
sale
25
5
aubainerie
sale
14
*Search Ads Popularity (SAP)shows the popularity of the search term from 5 to 99.

☑️Analyze current creatives and identify gaps

  1. Ask yourself – what creatives are you using at the moment?
    • images;
    • videos;
    • texts;
    • headlines;
    • CTAs (calls to action).
  2. Determine how well your current creatives resonate with your target audience. Collect data on the demographics, interests, and your audience behavior.
  3. Based on the data, determine which elements of your current creative are not working effectively. For example, if a certain headline has a low CTR, it should be replaced.
  4. Find areas where there is potential for improvement. New content formats, advertising channels, or approaches to audience segmentation are all points of growth.
  5. Based on the analysis of the current situation and identified gaps, create hypotheses that you plan to test. For example:
  • if the images don't grab enough attention, the hypothesis might be: “Brighter images with people will improve engagement.”;
  • if CTAs don't lead to conversions, the hypothesis might be: “Changing the CTA text into a more specific one will increase conversions.”.

☑️Define your testing goals

Clearly define the goals of testing. It can be an increase in conversions, a decrease in cost per click, a boost in engagement, etc. But it's worth remembering that even test creatives should be in line with the business concept.

☑️Select items for testing

Write down what creative elements you plan to use in the test: headlines, images, CTAs, text messages, page layouts, etc.

☑️Create multiple variants of creatives

Develop several versions of each element. For example, if you plan to test headlines, create a few variations of them.

☑️Define success metrics

Set the success metrics you want to measure for each option: CTR, conversions, cost per conversion, engagement rate, etc.

☑️Select a testing methodology

A/B testing (split testing), A/A/B or A/B/B testing, multivariate testing (MVT), Conversion Rate Increase tool.

☑️Conduct a test launch

Divide your audience into groups and start testing. Check and make sure that the test is conducted for a reasonable period and with the necessary amount of traffic to obtain the correct data.

☑️Analyze the data you receive

Gather and compare the results of the different options across the defined metrics. Analyze which options performed best.Based on the data, decide on the next stage of the action plan. For example, it can be either choosing the best option for use in campaigns or conducting additional testing of other hypotheses.

☑️Implement and optimize the system

Implement the best creatives. Constantly monitor their effectiveness.

☑️Keep on testing

Creative testing is an ongoing process. Don't stop and keep testing new ideas and approaches to continuously improve your campaigns and reach new peaks.

Device
Icon size
iPhone
180px × 180px (60pt × 60pt @3x)
120px × 120px (60pt × 60pt @2x)
iPad Pro
167px × 167px (83.5pt × 83.5pt @2x)
iPad, iPad mini
152px × 152px (76pt × 76pt @2x)
App Store
1024px × 1024px (1024pt × 1024pt @1x)
Icon format
PNG
Color models
sRGB or P3 (see "Color Management")
Layouts
Aligned without transparency
Sizes
Shape
Square without rounded corners
All the current requirements for icons are specified in the specification.

Five mistakes to avoid when creative testing

To avoid these mistakes, you can increase the effectiveness of creative testing and get more relevant results that will help you make a decision:

  1. Insufficient sample – running a test on a few users can lead to incorrect results. To determine the size, use the sample size calculator.
  2. Short testing period – conducting a test over a short period can lead to inaccurate results. It is necessary to set aside time for testing, taking into account the seasonality and behavioral changes of users. For example, the minimum duration of small tests for applications with a large volume of traffic can last from one to two weeks. For applications with less traffic, the testing time can be three to six weeks.
  3. Ignoring external factors – seasonality, competitors' marketing campaigns, and political, social, and economic changes can affect test results.
  4. Using low-quality creatives – testing low-quality creatives negatively affects the results and can lead to wrong outcomes.
  5. Lack of a post-test action plan – before you start testing, determine how you will use the results and what actions you plan to take based on them.
Icon format
32-bit PNG (with alpha channel)
Color models
512 х 512 pixels
Layouts
1024 КБ
Sizes
square, without rounding and shadows (Google automatically rounds the corners and adds shadows)
All the current requirements are specified in the specification.

To conclude, creative testing is a powerful tool for increasing the effectiveness of marketing campaigns. A structured and clear approach to testing, setting goals and metrics, proper audience segmentation, and analyzing the results allow you to make informed decisions.

As a result, you can optimize strategies to achieve better results. Avoid common mistakes, improve conversions and marketing campaign performance, reduce costs, and deliver a better user experience.

Home
Up