Pre-Planning
Q: What are the primary benefits of the MetaMarketing process?
A: The MetaMarketing process is used to increase customer inquiries and customer visits which leads to increased sales and profits. This is accomplished by determining in quantitative terms which factors are influential and by how much.
Q: Once we have optimized our ad, do we ever need do it again?
A: You would want to repeat the process when:
Factors or elements of the ad have changed
Market conditions have changed
Competition has caught up with your improvement
Q: Since we advertise in multiple media sources such as newspapers, radio, TV, direct mail and the internet, how do we decide which one would benefit the most from the MetaMarketing process?
A: Favorable answers to the following two questions help you decide the primary medium to apply the MetaMarketing process.
Can 4-18 experiments be carried out cost effectively and within time requirements?
Can the response from the advertisement in this medium be easily measured in quantitative manner?
Q: What kind of project can benefit from the MetaMarketing process?
A: Any advertisement design that poses multiple options due to presence of many factors/elements can benefit from this process. Advertisements using internet are generally accomplished in shorter time and are attractive for deriving benefits from MetaMarketing process.
Q: What do we need to do before we get started?
A: You need to select an advertisement that you would like to optimize. Then, select a team and appoint the leader of the team. This person is called team leader of project manager.
Q: How many people should be in a team and who make the best team members?
A: The team for MetaMarketing should have people with first-hand knowledge about the preparation of the advertisement and the target audience. A team members should be between 3 -10 people.
Q: What are the responsibilities of the team members and their time commitment needs?
A: Most of the team members have to participate in the experiment planning discussions where their ideas and input are captured. The team leader or involved team members will need to follow through with the recommended tests and see that they are done as planned. The planning session for MetaMarketing are generally held during a working day and is composed of three separate meetings facilitated by MetaMarketing specialist.
Q: Are the business owners and top management people required to be on the team?
A: The persons who authorize funds and are responsible for your advertisements should be on the team. These persons must attend the first of the three experiment planning sessions. This meeting usually takes 1 – 2 hours.
Q: When should we call to get started with the MetaMarketing process?
A: If you have your team ready, the team leader should reserve a day to hold the planning meeting with the help of MetaMarketing consultants. The team leader should contact MetaMarketing to establish the day’s schedule and determine who will be required to attend.
Q: Are there any special needs for the planning meeting?
A: Any one of the following three options will suffice:
Erasable white board with a supply of fresh dry erase markers
Clean chalk board with supply of chalk and a clean eraser
Easel with fresh pad and fresh markers
Planning Meetings
Q: How long do the planning meetings take and who should participate in them?
A: Experiment planning process is accomplished in the following three separate meetings.
Objective Definition – Top management and key team (including leader) should be present in this meeting. Medium of advertising and goal of the project in terms of measurable units are defined here. (1 – 2 hours)
Factors/Element Selection – This meeting is held with the whole team. The factors for experiments and their levels are first gathered by brainstorming process. The list of items are then qualified and reduced to a level that can be economically and time effectively carried out. (2 – 3 hours)
Execution and Data Collection – The team members who will be responsible for carrying out the experiments and provide resources should be in this meeting. The method for data collection, recording, and transfer are determined here. Any issues related to completing the experiments in timely manner are also discussed in this meeting. (1 hour)
Q: How many days should planning meetings take?
A: It is strongly suggested that the planning meetings be scheduled in the order shown above and that they are done all during one working day. If necessary, they can be done in separate days.
Q: Who decides which factors are to be studied and what the size of the experiment will be?
A: All decisions about the factors, levels and other issues of the experiment are made by the team consensus or top management. MetaMarketing consultants facilitate the decision making process.
Q: What happens after the planning meeting?
A: MetaMarketing consultants will document and take all the information from the planning meeting. This information is used to design the experiments.
Q: Is there any involvement or time commitment from the team members after the planning meeting?
A: Only the team leader or his/her designate will be involved in making sure the experiments are run properly. Most other team members will remain as information recipients unless involved in the preparation of the advertisements.
Q: Once the planning meeting is over, when can we expect receive the experiment designs?
A: MetaMarketing will deliver your experiment design complete with written instructions regarding data collection to you in 10 days or less.
Experiment Design
Q: What does the word “Design” mean in experiment design?
A: The task of design seeks answers to two questions:
How many experiments to do
What are the recipes of the experiments
Answers to both questions are obtained by laying out & designing the experiment following rules established by the experts in the field.
Q: What information is needed for designing experiments?
A: Elements and options. MetaMarketing utilizes a proprietary process to collect and select the list of elements for the study.
Q: What is the range of size of experiments?
A: Between 4 to 18 experiments depending on the number of factors and their options.
Q: What affects the size of the experiment?
A: Two items:
Number of elements
Number of options in each element
Q: How are the recipes in each experiment determined?
A: MetaMarketing uses the Taguchi experimental design technique which utilizes a predefined set of table of numbers called orthogonal array to layout the experiments. The arrays contain formula to distinctly describe the individual experiments.
Q: What is the largest number of elements that can be studied?
A: Theoretically, the limit is very high, say 63 elements. Most common experiments are designed to study 7 -15 elements.
Q: How many options can an element have?
A: 2 – 4 options. Most common studies involve 2 options for each element.
Q: What would be the size of experiments if we have up to 10 factors at 2 options each?
A: You will need to run 8 experiments if you include 7 elements with 2 options in each. The number of experiments will increase to 12 if the elements exceed 7 and up to 11.
Conducting Experiments
Q: When performing an advertising experiment, what is an acceptable sample size for each experiment?
A: The larger the sample size, the better the results. Following are some guidelines:
1,000 – 10,000 printed mails & flyers
5,000 – 50,000 for banner ads
100 – 500 for landing page optimization
3 – 20 for focus group or subjective evaluations
13 – 26 weeks for weekly newspaper ads
Q: What is the sequence of running the experiments?
A: The best way to run the experiments is in random order.
Q: If you have the option to run this experiment simultaneously, should it be preferred over running in sequence?
A: Running all experiments at one time produces better information. This may be the case for banner ad or landing page optimization by randomly selecting the experiment (design of ad). When all experiments are carried out simultaneously, the influence of uncontrollable factors does not affect the conclusions. Some experiments can not be run simultaneously such as weekly newspaper display ads.
Q: If each experiment is conducted at a different time, what should be the time gap between experiments?
A: This would depend on specific application. One must always be careful about timing the experiment such that influence of time of day, day of week, season in the year, etc. does not unduly influence one experiment over another.
Q: What do we do with the results after we carry out the experiments?
A: Collect results as prescribed and pass them on to MetaMarketing consultants.
Analysis of Results
Q: What does analysis of results produce?
A: Analysis of the planned experiments produces mainly the following information:
Optimum, most desirable design condition
Factor influences
Estimate of improvement from optimum condition
Q: In what form would we get the findings from analysis of results?
A: After analysis is completed, you will receive a written report containing optimum design and information about how to incorporate it in your mainstream design.
Q: How long does analysis take?
A: MetaMarketing will complete the analysis in 10 days or less.
Q: Is it possible for us to analyze the results ourselves?
A: Yes, but only if you have a person on staff with a background in the Taguchi design of experiment (DOE) technique.
Q: Would the detail calculation be available to us?
A: Yes. Details of analysis will be provided upon request.
Q: Is there software available for analysis?
A: A number of software packages are available in the open market. MetaMarketing utilizes Qualitek-4 software available from Nutek, Inc.
Q: What would be the content of a typical report of analyses?
A: A typical report will contain what you need immediately to proceed with the implementation and derive benefit. It includes: (a) Description of optimum design, (b) Recommendation for running confirmation test, and (c) any unusual events/findings.
Confirmation and Validation
Q: Why do we need to run confirmation tests?
A: Confirmation tests serve to verify the performance from the optimum design.
Q: How long or how many samples should we run the confirmation test?
A: Sample size/time varies depending on the type of test. A general rule of thumb is to run twice as many or as long as the individual experiments were tested.
Q: How would we know that we have confirmed or that the MetaMarketing experiment has been successful?
A: The following are key indicators.
Average performance of optimum conditions tested confirm improvement from previous established performance
Average performance of confirmation tests is better than the average of all experiments tested before
Average performance is within the confidence interval (predicted boundary or margin of error)
Q: Is it possible that the optimum condition does not show any improvement?
A: It is possible, but not probable. Under some extreme situations, misleading results may be obtained.
Q: What are some of the common extreme situations?
A: Following are the possible causes for not finding the improvement.
Effect of uncontrollable factors outweighs the beneficial effects of factor-option adjustments
Market condition and/or elements have changed from what was tested before
The current design condition is at optimum