Source and Purpose

Posted Date: 05/01/2020

Ted Carter
KASB Chief Data Officer
tcarter@kasb.org

I have written several pieces in recent months aimed to help consumers of information get a better feel for what is good research and what is bad.  We’ve talked about Averages, Deviations and Ranges, and Likert Scales.  In this article we will talk about looking at the source(s) of information and the intended purpose, whether it is stated or not.

No human being is without bias.  And, by extension, no organization created by humans is without bias.  Everyone has some kind of a stake in the research they do, and it is very, very difficult to ensure fair and impartial presentation of facts.  Further, we often hear of this information from second-hand sources that quote the original work, and these second-hand sources may be putting their own slant on what they report.

When reading a news story about some new finding, for example, it is important to consider all the sources by which you are receiving the information:

• The news outlet who is presenting the story: Are they considered relatively unbiased, or is there a known slant to what they report?  Several groups such as allsides.com and adfontesmedia.com provide lists of large news outlets and their apparent biases, but some argue that these sources are also biased.
• The author who wrote the story: Not every staff person within an organization has the exact same perspective as the company they work for.  Reporters and journalists all have their own beliefs and values, and only the really good ones are able to put forth work that is not notably swayed by these factors.  In addition, reporters and journalists are not necessarily experts on the research they are reporting on, so they may not fully grasp the information they are presenting.
• The publication where the research initially appeared: Did the research report initially appear in a peer-reviewed scientific or academic journal?  Was it published by the organization that funded the research?  Did it appear in a textbook?  Magazine?  Where the work first appears can tell you a lot about its potential bias.
• The organization under which the research was conducted: Was the research done by a university?  By a nonprofit with a political aim?  By an organization with a known bias?  The company that pays for the study can have a big impact on the information presented and the conclusions reached.
• The person(s) who wrote the report: Is the author(s) a researcher or statistician?  Were they a lobbyist or public relations person?  Was it written by an individual or by a team?  Was it written by the person(s) who actually conducted the research or did the analysis?  Sometimes a research brief or summary is written by someone other than those that did the research, and some inaccuracies can come just in the process of summarizing.
• The person(s) who conducted the study or analysis: Were the people doing the study, conducting the experiment, or performing the data analysis trained primarily in those activities?  Do their job titles include things like advocacy or communications or PR?  Did they have any motivations for presenting biased or selective information?

In order to have a piece of information that is virtually doubt-proof, you might look for research that is:

• Conducted by a researcher or statistician.
• Written by the person conducting the research.
• Sponsored or conducted under an organization known to be without substantial bias and/or focused on research and presentation of unbiased information.
• Published in a peer-reviewed and well respected scientific or academic journal.
• Reported by an author or journalist known to have a high level of integrity and fairness of reporting and who has adequate understanding of the subject matter and methods used.
• Reported in a publication known to have a high level of integrity and fairness of reporting.

But what about the vast amount of information in the world that does not meet all these criteria?  Should you just ignore it?  And if not, what else should you consider?

Let’s use the information KASB provides as an example.

Most of the work called “research” or “analysis” produced by KASB is done by one of two people – myself or Mark Tallman.

Let’s start with me.  I have a background in research and analysis, and my previous title was “Research Specialist.”  My current title, “Chief Data Officer,” still indicates that my primary function is all about information.  However, I have worked in the education field almost all of my adult life.  Both of my parents were educators.  Both of my children have come up through Kansas’ public school system, and I have always been a strong supporter of public schools.

Then there is Mark.  He has been a lobbyist for KASB for a long time.  His title is “Director of Advocacy, Communications, Research, and Marketing.”  He oversees all of KASB’s advocacy efforts.  He is a former public school board member.  Most of his career has been spent supporting and promoting public schools.

And we both work for an organization that’s stated mission is to “provide a culture of collaboration and service, be a voice of public education, and improve student education outcomes.”

So, should you discredit everything we say?

Obviously I am going to answer “no” to that one, but I think I can back it up.

One of the things I have alluded to a few times that does not tie directly to the education, background, or primary job functions of the people involved or the overall purpose and goals of the organizations involved is simply integrity and reputation.

KASB has a strong reputation for presenting information fairly.  We work hard to cite our sources and note the limitations of the conclusions we can draw.  We try to acknowledge when there are gaps in what we can “prove” as “fact” and what we can only point to as an indicator of what is most likely the case.

Mark Tallman has a reputation with KASB members for being a reliable source of information.  But more importantly, he has reputation for providing useful information by legislators, the Governor’s office, and other stakeholders throughout Kansas.

I know my reputation is not as strong as Mark’s, but hopefully I have built some credibility over my years here as someone who tries to present information in an unbiased fashion.

Nonetheless, there are those who would discredit or ignore anything produced by myself, Mark, or KASB in general simply because our primary goal is to support Kansas public schools.

As I said at the beginning of this article, no human is without bias, and by extension no organization is either.  The important thing for consumers of information to be able to do is identify the potential sources of bias and keep them in consideration when reading and evaluating research.

Here are some steps you can take:

1. Consider all the people and organizations involved in presenting the information as you receive it – their backgrounds and expertise, their primary purposes, and their reputation for presenting accurate and complete information.
2. If you are getting the information from a second-hand source, see if you can get to the original source of the research – the initial report where the findings were presented. Further, if there is a high-level summary and a full report available, take some time looking at the full report to see if the summary accurately reflects everything included in the full report.
3. If the organization has a known bias, see if you can find other sources for the same kind of reporting. It is sometimes even helpful to find an organization that has a known bias in the opposite direction and compare the kinds of conclusions they draw from available data and research.

And finally, it is important to consider your own biases, background, and expertise, and how it might influence how you interpret the information you are receiving.