In the late 1970s and early 1980s, there was a popular game show on television called “Family Feud.” In the show, two families were pitted against each other answering questions that had been answered by 100 other people. Points are awarded based on the popularity of the answer as compared to the 100 people polled. The questions are not based on reality or truth, but rather on what other people think is true. When a contestant gave an answer, the game show host would say, “Survey Says…” As the official report of the BRTFSSG due in mid-October 2009 is being prepared for the President’s Office, the people of the Missouri Synod need to be asking what the “survey says.”
The October 2009 Reporter indicates that the BRTFSSG will be using input from convention delegates to form the final report. After this report is presented to President Kieschnick and after convention delegates are selected, the President will Restructuring Plan. No doubt the “collective will” of the Missouri Synod as expressed in what the “survey says” will be an important component. Before considering what the “survey says,” some background on the survey and its methodology might be helpful.
Recently, the BRTFSSG released the survey results of the survey given at each of the 35 district conventions. (Survey Results available here) According to the website, the goal of the survey was to give the BRTFSSG a “sense of the assembly” to each of the proposals. The website has a statement about the “reliability” of the survey which says, “They represent an accurate gauge of those delegates’ sentiments regarding the items presented.” Therefore, according to the BRTFSSG, the “survey says” represents the “collective will” of the districts of the LCMS. Anyone questioning or disagreeing with the restructuring proposal is going against the will of the people of the Missouri Synod. What the “survey says” ends up being very important for how the restructuring will be viewed by the convention delegates.
Let’s consider for a moment how these surveys were administered. At the first district convention held in North Dakota, the survey was distributed just before the BRTFSSG presentation. The survey only was given to convention delegates, meaning that others in the room were not included in the polling. As soon as the BRTFSSG presentation was completed, the surveys were collected. Once the surveys were collected, there was a question and answer session about the BRTFSSG’s presentation. Several delegates who had answered positively to the survey during the BRTFSSG’s presentation, changed their mind after the question and answer session; however, the surveys were collected and no changes could be made. Whatever one thinks of the manner in which the survey administration was handled in North Dakota, for the survey data to have any meaning, the method of administration would have to remain exactly the same for the remaining 34 district conventions. However, that is not what happened.
At the next district convention held in Southern Illinois, the survey administration methodology was changed. At both North Dakota and Southern Illinois only convention delegates could take the survey. Like North Dakota, the survey was distributed just before the BRTFSSG’s presentation. However, in Southern Illinois, it was not collected until after the question and answer session. This gave the delegates the opportunity to base their survey responses on how their questions about the restructuring were answered. This seems like a better or perhaps more objective survey methodology than that followed in North Dakota. (Perhaps, the President’s Office was bowing to criticism that people did not have the opportunity to respond to the survey after asking questions.) In any case, the methodology of the survey administration changed between the first and second district conventions held. While we are not experts in survey construction or administration, any credible survey that ensures reliable results would not alter its methodology like this. The results of the first two surveys are already rendered meaningless.
While we have not been able to determine when, that is, at which district convention, the next major shift in survey administration methodology occurred, there was a point after which the survey was given to not only delegates but any breathing, warm body in the room. The BRTFSSG declared that it was interested in everyone’s opinion regarding the restructuring. For all the BRTFSSG knows, the convention hall and hotel staff were taking the survey at these district conventions. Perhaps an individual filled out 10, 20, or 50 surveys. Whether these sorts of things happened or not, no one knows, especially not the BRTFSSG.
With the looser administration of the survey, people were allowed to turn in the survey before, during, or after the BRTFSSG’s presentation. How many surveys were turned before people heard the question and answer? No one knows. How many people would have responded differently after hearing the question and answer sessions? No one knows. How many people filled out a survey before or during the presentation and turned it in, only to grab another survey and fill in different answers after the question and answer session? No one knows. The point is that the survey results are completely unreliable, unscientific, and in the end meaningless. No one can say what the survey actually captured. Considering that the Synod paid nearly $12 million to consultants for the Ablaze and Fan into Flame campaigns, one would think that the President’s Office could hire a competent consultant to advise and to administer a survey in a way that produced meaningful results. What the “survey says” cannot be trusted or considered representative of the Synod as a whole.
We have not even begun to analyze the survey questions themselves. For instance, who among anyone that loves the Missouri Synod (most of our pastors and church members) are going to answer negative to clarifying “mission and purpose of the Synod”? These sorts of leading questions are tantamount to asking, “Do you love your mother?” Who is going to say, “No”? The questions themselves often produced meaningless results. Again, a President who can spent $12 million on consultants for a failed fund raising campaign surely could have spent a few dollars on competent survey consultants. Then perhaps the goal of the survey wasn’t to really find out what the Synod truly thought about the restructuring but rather to guarantee that the President and his blue ribbon task force obtained the answers they desired.
We need to keep in mind that on Family Feud the survey was not designed to measure truth or reality but what a certain group of people thought. As a result, completely rational, logical, and true answers contestants gave on the show Family Feud received the buzzer. All that mattered was answering the way a select group of people answered. If 75 of the 100 answered that the sun was “purple” and a contestant said the sun was “yellow,” then the contestant was wrong, likewise, the BRTFSSG’s survey.
Ironically, the survey results do not agree with district overtures regarding the President’s restructuring proposal (let us never forget that the BRTFSSG was appointed by President Kieschnick and that he felt the Synod needed to be restructured). For instance, the Southern Illinois District passed several overtures in their convention to reject and delay the President’s restructuring proposal. Yet ironically, the survey results so that the Southern Illinois District is remarkably in favor of restructuring. Depending on which restructuring proposal, the Southern Illinois District approves the restructuring by 50% to 80%. Amazing how a district can pass several overtures against the President’s restructuring proposal yet the “survey says” the Southern Illinois District approves the restructuring. This schizophrenic dichotomy exists for all the districts that passed overtures against the restructuring. How can a district that passed a anti-restructuring overture by 90% to 10%, “survey says” approve of the restructuring by 73%? My friends there truly is trouble in “River City” as Professor Harold Hill (President Kieschnick) swindles the entire Missouri Synod with his “survey says.”
“Survey says” may well be a way for President Kieschnick to bully the delegates elected to the 2010 Synod Convention. How dare the convention delegates question the “collective will” of the “survey says”? After all, the 7,649 survey respondents “provide a reliable gage of those delegates’ sentiments regarding the items presented.” The President’s blue ribbon task force’s own disclaimer regarding the survey, “since these delegates are not statistically representative of all members of LCMS congregations, they are not generalizable to that group,” demonstrates that the survey results are utterly meaningless.
Whether the Synod will be swindled by “survey says” is yet to be seen. May Christ grant the 2010 Synod Convention Delegates the wisdom of Solomon to divide the truth from the “lies, damn lies, and statistics” of the President’s Blue Ribbon Task Force’s “survey says.”