“Survey Says . . .” by Publius Aequillus

In the late 1970s and early 1980s, there was a popular game show on television called “Family Feud.” In the show, two families were pitted against each other answering questions that had been answered by 100 other people. Points are awarded based on the popularity of the answer as compared to the 100 people polled. The questions are not based on reality or truth, but rather on what other people think is true. When a contestant gave an answer, the game show host would say, “Survey Says…” As the official report of the BRTFSSG due in mid-October 2009 is being prepared for the President’s Office, the people of the Missouri Synod need to be asking what the “survey says.”

The October 2009 Reporter indicates that the BRTFSSG will be using input from convention delegates to form the final report. After this report is presented to President Kieschnick and after convention delegates are selected, the President will Restructuring Plan. No doubt the “collective will” of the Missouri Synod as expressed in what the “survey says” will be an important component. Before considering what the “survey says,” some background on the survey and its methodology might be helpful.

Recently, the BRTFSSG released the survey results of the survey given at each of the 35 district conventions. (Survey Results available here) According to the website, the goal of the survey was to give the BRTFSSG a “sense of the assembly” to each of the proposals. The website has a statement about the “reliability” of the survey which says, “They represent an accurate gauge of those delegates’ sentiments regarding the items presented.” Therefore, according to the BRTFSSG, the “survey says” represents the “collective will” of the districts of the LCMS. Anyone questioning or disagreeing with the restructuring proposal is going against the will of the people of the Missouri Synod. What the “survey says” ends up being very important for how the restructuring will be viewed by the convention delegates.

Let’s consider for a moment how these surveys were administered. At the first district convention held in North Dakota, the survey was distributed just before the BRTFSSG presentation. The survey only was given to convention delegates, meaning that others in the room were not included in the polling. As soon as the BRTFSSG presentation was completed, the surveys were collected. Once the surveys were collected, there was a question and answer session about the BRTFSSG’s presentation. Several delegates who had answered positively to the survey during the BRTFSSG’s presentation, changed their mind after the question and answer session; however, the surveys were collected and no changes could be made. Whatever one thinks of the manner in which the survey administration was handled in North Dakota, for the survey data to have any meaning, the method of administration would have to remain exactly the same for the remaining 34 district conventions. However, that is not what happened.

At the next district convention held in Southern Illinois, the survey administration methodology was changed. At both North Dakota and Southern Illinois only convention delegates could take the survey. Like North Dakota, the survey was distributed just before the BRTFSSG’s presentation. However, in Southern Illinois, it was not collected until after the question and answer session. This gave the delegates the opportunity to base their survey responses on how their questions about the restructuring were answered. This seems like a better or perhaps more objective survey methodology than that followed in North Dakota. (Perhaps, the President’s Office was bowing to criticism that people did not have the opportunity to respond to the survey after asking questions.) In any case, the methodology of the survey administration changed between the first and second district conventions held. While we are not experts in survey construction or administration, any credible survey that ensures reliable results would not alter its methodology like this. The results of the first two surveys are already rendered meaningless.

While we have not been able to determine when, that is, at which district convention, the next major shift in survey administration methodology occurred, there was a point after which the survey was given to not only delegates but any breathing, warm body in the room. The BRTFSSG declared that it was interested in everyone’s opinion regarding the restructuring. For all the BRTFSSG knows, the convention hall and hotel staff were taking the survey at these district conventions. Perhaps an individual filled out 10, 20, or 50 surveys. Whether these sorts of things happened or not, no one knows, especially not the BRTFSSG.

With the looser administration of the survey, people were allowed to turn in the survey before, during, or after the BRTFSSG’s presentation. How many surveys were turned before people heard the question and answer? No one knows. How many people would have responded differently after hearing the question and answer sessions? No one knows. How many people filled out a survey before or during the presentation and turned it in, only to grab another survey and fill in different answers after the question and answer session? No one knows. The point is that the survey results are completely unreliable, unscientific, and in the end meaningless. No one can say what the survey actually captured. Considering that the Synod paid nearly $12 million to consultants for the Ablaze and Fan into Flame campaigns, one would think that the President’s Office could hire a competent consultant to advise and to administer a survey in a way that produced meaningful results. What the “survey says” cannot be trusted or considered representative of the Synod as a whole.

We have not even begun to analyze the survey questions themselves. For instance, who among anyone that loves the Missouri Synod (most of our pastors and church members) are going to answer negative to clarifying “mission and purpose of the Synod”? These sorts of leading questions are tantamount to asking, “Do you love your mother?” Who is going to say, “No”? The questions themselves often produced meaningless results. Again, a President who can spent $12 million on consultants for a failed fund raising campaign surely could have spent a few dollars on competent survey consultants. Then perhaps the goal of the survey wasn’t to really find out what the Synod truly thought about the restructuring but rather to guarantee that the President and his blue ribbon task force obtained the answers they desired.

We need to keep in mind that on Family Feud the survey was not designed to measure truth or reality but what a certain group of people thought. As a result, completely rational, logical, and true answers contestants gave on the show Family Feud received the buzzer. All that mattered was answering the way a select group of people answered. If 75 of the 100 answered that the sun was “purple” and a contestant said the sun was “yellow,” then the contestant was wrong, likewise, the BRTFSSG’s survey.

Ironically, the survey results do not agree with district overtures regarding the President’s restructuring proposal (let us never forget that the BRTFSSG was appointed by President Kieschnick and that he felt the Synod needed to be restructured). For instance, the Southern Illinois District passed several overtures in their convention to reject and delay the President’s restructuring proposal. Yet ironically, the survey results so that the Southern Illinois District is remarkably in favor of restructuring. Depending on which restructuring proposal, the Southern Illinois District approves the restructuring by 50% to 80%. Amazing how a district can pass several overtures against the President’s restructuring proposal yet the “survey says” the Southern Illinois District approves the restructuring. This schizophrenic dichotomy exists for all the districts that passed overtures against the restructuring. How can a district that passed a anti-restructuring overture by 90% to 10%, “survey says” approve of the restructuring by 73%? My friends there truly is trouble in “River City” as Professor Harold Hill (President Kieschnick) swindles the entire Missouri Synod with his “survey says.”

“Survey says” may well be a way for President Kieschnick to bully the delegates elected to the 2010 Synod Convention. How dare the convention delegates question the “collective will” of the “survey says”? After all, the 7,649 survey respondents “provide a reliable gage of those delegates’ sentiments regarding the items presented.” The President’s blue ribbon task force’s own disclaimer regarding the survey, “since these delegates are not statistically representative of all members of LCMS congregations, they are not generalizable to that group,” demonstrates that the survey results are utterly meaningless.

Whether the Synod will be swindled by “survey says” is yet to be seen. May Christ grant the 2010 Synod Convention Delegates the wisdom of Solomon to divide the truth from the “lies, damn lies, and statistics” of the President’s Blue Ribbon Task Force’s “survey says.”


Comments

“Survey Says . . .” by Publius Aequillus — 10 Comments

  1. Collecting the surveys after the presentation and before any other information would only tell you how convincing they were with how they presented the proposals, not how many people approved of them. If they are asking such leading questions and only explaining the benefits while ignoring any potentially negative consequences it is not surprising to see high numbers. One would wonder why such a lopsided and biased survey didn’t produce even higher numbers.

    One thing I try to remember in all of this is that the final proposals aren’t even out yet. The surveys changed from one place to another. The numbers don’t mean anything. The only benefit from the surveys would have been any comments to help the task force fine tune the proposals, or at least how they present them. I believe that was the rationale given for the surveys, to seek feedback, not to poll approval. Any inferences about people approving or disapproving anything are meaningless. Nothing can indicate the approval of proposals before they have been made. Only after the final proposals are released will there be anything of consequence to discuss on the matter.

    The post above shows that any statistical information from the surveys are meaningless and that is all that has been released. If they want to share meaningful information from the surveys they would need to produce the surveys given at each convention and the comments they received from each convention. That would at least help to show how the proposals were shaped. In the end, how the proposals were shaped is not as important as what the proposals actually are, though knowing how they took shape would help in understanding what the authors meant and how they are likely to be interpreted.

  2. This schizophrenic dichotomy exists for all the districts that passed overtures against the restructuring.” It’s not schizophrenic at all. The BRTFSSG asked ambiguous questions that didn’t sound bad on the surface. But district conventions passed CLEAR AND UNAMBIGUOUS RESOLUTIONS with whereases and therefores. If they didn’t know what there were being asked of the Blue Ribbon Task Force, they did know what they were saying with their resolutions.

    Explicit resolutions trump questionable surveys any day of the week and twice on Sunday.

    I agree with Mollie that the survey results (as well as the “regional gatherings”) could easily be used by Synod, Inc. to “poison the well” so that anyone who raises reasonable concerns or objections is going against the collective will of the district convention delegates: dissenters want the church to be a dying museum that cares nothing for evangelism. Demonize the opposition. Where have I heard this before?

    RULE 12: Pick the target, freeze it, personalize it, and polarize it.” Cut off the support network and isolate the target from sympathy. Go after people and not institutions; people hurt faster than institutions. (This is cruel, but very effective. Direct, personalized criticism and ridicule works.)

  3. In the NID survey, on May 3, roughly 1/3 of the way into the convention season, we were read the survey question and then asked to respond within about 50 seconds and then the next question was read.

    For those of you who have taken the online survey of the Interested Laymen you can see the results and will notice that they differ drastically from the synod’s survey. The Laymans’ survey was was done far more scientifically and which had both pro and con for each question. I will post a comparison later this week. To take the Layman’s survey go to http://blueribbonsurvey.org/index.php

    TR

  4. “Then perhaps the goal of the survey wasn’t to really find out what the Synod truly thought about the restructuring but rather to guarantee that the President and his blue ribbon task force obtained the answers they desired.”

    Corporations and politicians do this all the time. Wouldn’t surprise me in the least.

  5. Is there any question that Synod admin wnats to pass these proposals? Were there any proposals submitted by the “blue ribbon” task force that were rejected? If so, I havn’t heard of them. The deck has been stacked.

  6. Are we going to abide by our district and synodical constitutions, or are we going to permit survey results (or worse yet, the results the president’s office chooses to report) to decide matters among us?

    Surveys don’t express the will of the Districts — District convention resolutions do.

    It would be a further trampling of our synodical and district constitutions if the 2010 delegates listened to dubious district surveys, while disregarding the express will of the districts’ convention resolutions regarding President Kieschnick’s restructuring plan.

    TW

  7. ““Survey says” may well be a way for President Kieschnick to bully the delegates elected to the 2010 Synod Convention.”

    Bully? Please,… President K doesn’t bully – he harangues, threatens with a swarm of locusts, water turning to blood and the death of all the first born. Well, maybe not the last. But you can be sure that he will use the survey to his heavy-handed advantage on those bedazzled with slick presentations, moving testimonials from celebrity folk – (remember Miss America is LCMS this year) and his own version of “Chicken Little’s,” “The Sky is falling, the sky is falling!” Want to bet that the resolution nixing the BRTFSSG’s proposals won’t even receive a reference in the Resolutions except for a sop?

  8. Anonymous–I think the “Rule No. 12” you quoted is from Saul Alinsky. It’s a favorite of Chicago politicians, but not limited to them, or their party.

  9. The results of the survey cannot be trusted because of the conflicting methodologies employed by the Task Force.

    Relying on the survey as an accurate measure of the delegates’ opinions implies objectivity, that those taking the survey were making informed decisions based on all the available facts, and that those conducting the survey did so from a position of neutrality and impartiality.

    The Task Force, by its own admission, did not present the material neutrally or objectively, but instead advocated for the proposals (i.e., for one side of the argument). Thus their surveys cannot be considered objective.

    If they wanted an accurate survey, they would have had make an objective presentation, considering both pros and cons without advocating for either side. If they wanted to abvocate for one side or the other (which they were free to do), it disqualifies them from conducting an objective survey. They can’t have it both ways.

    The bureaucratic response to the laymen’s survey illustrates that this principle is clearly understood. As those conducting the survey were assumed to be partisan against the proposals, their survey was consequently assumed to be inaccurate (biased against the proposals), and officially rejected for that reason. Meanwhile, the survey conducted by those partisan in favor of the proposals is not only accepted, but promoted–an interesting double standard.

    The Task Force’s official status doesn’t automatically make their survey accurate or objective. It’s just another push poll.

  10. As Pastor Russow pointed out above (#3), not only did they advocate for the proposals openly, the methodology of their presentation was such that it vertually guarenteed that they got favorable responses. They like to claim that it was objective because they allowed the survey takers to hand them in after the discussion period (in our case in the Missouri District we had until the next morning), but the presenter paused after each presentation point and virtually insisted that the survey takers fill out the survey then, before and discussion could take place.

    When the discussion period started, I stood up and denounced the disingenousness of the process and asked the other delegates to throw away their completed forms if they’d already filled them out, and wait until after the discussion period to fill out new ones. That earned me a very stern rebuke by our district president as well as some additional politicking by our synodical president before the discussion was allowed to continue.

    It was obvious to anyone there that they weren’t interested in an objective response.

    Eric Ramer, Interested Layman

Leave a Reply

Your email address will not be published. Required fields are marked *

Notify me of followup comments via e-mail. You can also subscribe without commenting.