On January 29 I posted a blog titled, “Poll Position.” It documented how, over the decades, public support for the government space program has been tepid at best.
Which is why I was suspicious when a poll released by a group called Explore Mars released a poll February 12 claiming that 75% of Americans want to increase NASA's budget to 1% of the federal budget, and 71% believe humans will go to Mars by 2033.
According to the press release:
The poll found that 71 percent of Americans are confident that humans will go to Mars by 2033. When told that there are currently two operational NASA rovers on Mars, 67 percent of respondents agreed the U.S. should send both humans and robots to Mars.
Americans, on average, believe that NASA spending represents 2.4 percent of the federal budget, with a standard deviation of 1.68 percent. In reality, the Administration’s request for NASA for FY2013 was $17.7 billion representing approximately 0.5 percent of the federal budget.
After being presented with this percentage, 75 percent of Americans said they “Strongly Agree” or “Agree” that it is worthwhile to increase NASA’s percentage of the federal budget to 1 percent to fund a mission to Mars.
Having dabbled in political consulting for many years, and having taken several college-level statistics courses over the years, I was suspicious of this poll's claims so I dug further.
The first thing you check with a poll is the population sample. Who was polled?
If my question was, "Should Barack Obama be re-elected?" and I polled only Republicans, obviously the answer would be skewed. The same if I asked only Democrats.
How you conduct the survey is also important. Telephone surveys are no longer considered reliable, because who has a phone, who answers and what type of phones are called (home, business, cell) all affect the poll's demographics.
So who was polled for this Mars survey?
According to the press release:
The survey was conducted by email and targeted a nationwide sample. All efforts were made to ensure a representative sample of the U.S. population 18 years and older given normal standards of statistical sampling.
E-Mail?!
That's even less reliable than a phone poll.
A Preliminary Snapshot Report of the poll is on their web site. It provides no more details about the poll's demographics, other than to state, “The full report, including demographic data, is scheduled to be released on March 4, 2013.”
So they won't even tell us who they polled until next month.
To further strain this poll's credulity, it was conducted by Phillips & Company. Visit their web site and you find out it's not a polling business. It's a marketing firm. To quote from their About page:
Phillips & Company is a global communications firm that helps clients create, defend and sustain leadership positions through public relations and business development.
We help companies and organizations focus on the opportunities and strategies that accelerate market adoption and success by creating greater demand for products, services and ideas. Ultimately, we help our clients build and reinforce their position as a trusted leader.
To further erode their credibility, Phillips & Company President Rich Phillips joined the Explore Mars Board of Directors in December. To quote from their press release:
"We are extremely happy to welcome Rich to the board of directors," said Explore Mars Executive Director Chris Carberry. "Rich's background and his ability to bring people together are precisely what Explore Mars needs at this point in time to help move the organization forward and effectively advance the cause of sending humans to Mars" ...
"The Explore Mars team has been instrumental at advancing the cause for exploring the Red Planet and the benefits to U.S. leadership and mankind," said Phillips. "I am confident that a man or woman will step onto the surface of Mars in our lifetime, and I am proud to work with Explore Mars to help fulfill that mission."
This business relationship and conflict of interest is not disclosed in the report.
So we have a feel-good poll conducted by e-mail of an unknown population sample, conducted not by a professional polling firm but by a marketing company whose president is on the client's board of directors.
This would get you punted out of a beginning Statistics course.
Someone at NASA bought into all this, because it was retweeted by @NASA without checking the poll's validity.
I'm all for human spaceflight and exploring Mars when it becomes financially and technologically viable, but let's not rig false polls to mislead the space advocacy movement into thinking there's widespread public support when there isn't. It only harms the cause, and will be dismissed by politicians who know all about polling and how to create a fake one.
The poll was a random sample of 1,101 indiviudals. The panel was randomly built using the riversampling methodology and we used a platform from Qualtrics and an independent team from there to build the panel. There was no manipulation and we did not at any time oversample from any population.
ReplyDeleteI understand your skepticism. Everything you stated about Phillips & Company's relationship with Explore Mars is true. I recently joined the Board and I do support their goals. However, we had no expectation of the results. When they came in, were were pleasantly surprised. But the results could have been very different and we would have reported it.
For each of the 1,101 respondents, we have collected data by zip code, income level, education level, ethnicity and age. But we haven't had time to compile that data in crosstabs to see if there are differences.
I greatly appreciate your skepticism. We need to challenge polls, especially those that purport results that are clearly strong. But I want to assure you that the poll adhered to a strict and statistically sound methodology. I have been conducting scientific polls all of my life and studied under some strong statisticians at Georgetown when I received my Masters in Public Policy back in 93.
You and your readers can trust the results. What we do with them is our next decision.
If your readers want to view the preliminary report they can download it at: http://www.exploremars.org/mars-generation-survey-first-results
ReplyDeleteThe methodology is explained and you can see the detailed breakdown by respondent.
Mr. Phillips, thank you for taking the time to respond.
ReplyDeleteI still have concerns. My primary concern is the perceived conflict of interest. As a member of the Explore Mars board of directors, you have an interest in seeing a favorable result to the poll, as it helps the corporation in its fundraising solicitation. The web site has a Donate button suggesting donations of $100, $200 or $500 with various goodies in return.
As for river sampling, by no means is that considered a bulletproof means of polling. If interested, folks can Google "river sampling market research" to find various opinions on the technique.
In my opinion, e-mail simply isn't credible because you don't know who's on the other end. You can do all the cross-research you want to create a profile of the individual you think is at that address, but you really don't know who it is. I have friends where both the husband and wife share the same e-mail address.
And as I mentioned in my original post, there have been far too many polls over the decades going back to the 1960s showing that public support of government spaceflight is tepid at best. I do think there's some validity to asking respondents what they think is the NASA budget, then telling them what it really is, and seeing how they react.
But at the same time, I've written many times that simply throwing more money at NASA doesn't magically translate into Starfleet. Congress determines what the money will be used for, and typically that's for more pork in their states and districts. If NASA's budget were doubled today to $35 billion (which won't happen), what would the money be used for? I guarantee you there would be a fist fight not only among the members of Congress, but also the various space advocacy groups. Should it go to human spaceflight? Robotics? Deep space probes? Aeronautics? How much goes to each?
And let's not overlook NASA's sad history of budget overruns. Some of that is due to Congress underfunding NASA, but NASA has also been guilty of mismanagement. Do you want to give a lot more money to a bad manager? I wouldn't.
The issue is far more complex, and an ethically comprised poll isn't going to change that.
I sympathize with the cause, but I still believe you folks took the wrong approach. It won't work, other than maybe raising more money for Explore Mars.