Research Fables from the Sisters Grinn, No. 11
Chicken Little was an eager young hatchling on a farm near Scholarship Forest, the home of Little Red Research Student. Little Red was attending graduate school 1 out of town, but by chance 2 was home visiting her trans-species Little cousins when Chicken Little was hatched. Many families of Scholarship Forest held Little Red up as a role model to their children, but Chicken Little actually imprinted on her. As a result, although poultry rarely aspire to scholarly careers, Chicken Little wanted to become an evidence-based health care provider when she grew up. She tried to practice research utilization faithfully every day.
One day, Chicken Little was walking in Scholarship Forest, seeking nuggets of knowledge 3 Without warning, something hit her on the head and landed at her feet. It was a research study, entitled "The Relationship of Relaxation and Pain." Chicken Little was intrigued. Pain management was a significant health care problem, and she was eager to read the latest research in the area. She read the abstract, and was surprised to learn that the findings indicated a positive relationship between relaxation and pain -- in other words, the more relaxed the subject was, the more pain the subject reported.
As was her habit 4 Chicken Little skipped immediately to the discussion section of the article. The researchers concluded that enhanced relaxation caused heightened pain perception and questioned the clinical benefits of relaxation therapies for patients with pain. Chicken Little was alarmed. Surely, the research must be important, or it wouldn't have been published! Clutching the study, she set off to alert her neighbors in Scholarship Forest.
Chicken Little had not gone long before she encountered Peter Possum, an aspiring sleep researcher. "Oh, Peter," began Chicken Little, "have you heard? Relaxation therapy makes pain seem worse, instead of better! We must warn the clinicians!"
She showed Peter the study, and Peter, too, read the abstract and discussion sections. "Well, I don't have much experience in the area, but we surely don't want to make anybody's pain worse. You know who needs to know? Roseanne Rabbit. She teaches relaxation therapies in her childbirth classes."
So Chicken Little and Peter Possum set off in search of Roseanne Rabbit. They found her at home, preparing her computer-generated visuals for her class that evening. "Look, Roseanne!" said Chicken Little. "This research says that relaxation therapy makes pain worse!" Chicken Little handed Roseanne the study, and she, too, read the abstract and the discussion sections.
"Oh dear!" cried Roseanne. "I thought I was doing something helpful for pregnant couples, and now it looks like it's not. I'll quit right away. And we need to warn Douglas Dog 5 who's just finishing up a textbook on health care interventions." Pausing only to save her slides to disk, Roseanne joined Chicken Little and Peter Possum on the path to the Dog house.
Douglas Dog was not at home when Chicken Little, Peter Possum, and Roseanne Rabbit arrived, but he appeared from the other direction very shortly after. "Good friends! What timing! I've just sent the draft of my new textbook to my publisher. Come in and join me in a celebration!"
"We can't stay, because we have to warn everybody. There's a new research study that says relaxation therapy causes pain!" said Chicken Little. She handed over the study for Douglas Dog to review. He read only as far as the abstract before turning pale and putting it down.
"Excuse me," said Douglas Dog, "but I have to leave right away. There might just be enough time to catch the overnight express service before they pick up my manuscript . 6 Chapter Five will have to be rewritten!" He hurried off.
Chicken Little was proud of her research dissemination efforts thus far, but she knew there were many other residents of Scholarship Forest who were still dangerously unaware of the newest research finding. How could she get to all of them in time, before another person was hurt by relaxation therapy? "I know! Let's go tell the local newspaper! That way, everybody will know right away, whether they have time to read the study or not!"
The group was well on its way to the local newspaper office, when they encountered Samantha Student. Samantha was Little Red's younger sister, and she was working as a forest maintenance academic between semesters. When she saw the group, she stopped pruning the undergrowth 7 and greeted them.
"Oh, Samantha," said Chicken Little. "Have you and Little Red heard? Relaxation therapy is bad for you! Look at this new study! We're on our way to tell the entire forest!"
Samantha put down her lopping shears and read the study abstract. Then, while Chicken Little and her friends waited anxiously, she read the introduction, the literature review, the research questions, the methods section, the findings, and the discussion section. Then she went back and read all the sections again. Finally, as Chicken Little hopped around her impatiently, she reread the findings. "Chicken Little, have you and your friends read the entire study?" asked Samantha.
"Well, no! We skimmed the most important parts, and thought we needed to let everybody else know right away."
Samantha rolled her eyes. "The most important parts ....! Okay, Chicken Little. Sit down right here and read through the entire report carefully. If there's anything you don't understand, ask me."
With Peter Possum and Roseanne Rabbit reading over her wings, Chicken Little read every word of the research report. It took a long time, and the sun was well toward sunset when they finished.
"Now," said Samantha. "What sample was studied in this research?"
Peter Possum sneaked another look at the methods section. "They studied left-pawed albino hamsters."
"It's been a while since I saw any of them in this forest," remarked Samantha. "So who in this forest do you think the findings might apply to?"
Chicken Little hung her head. "But they studied 10,000 left-pawed albino hamsters. Doesn't large sample size support broader generalization?" asked Peter.
"Only to the population sampled, which is still restricted to left-pawed albino hamsters," replied Samantha. "Now what kind of study did the researchers conduct? Was it an experiment?"
Chicken Little reread the methods section. "They say it's a descriptive study. They measured relaxation and pain without intervention in a randomly-selected sample of hamsters. But doesn't random selection of the sample make it an experiment?"
"No, Chicken Little. You're confusing random assignment of the sample and random selection 8 Since the researchers didn't manipulate levels of relaxation, how can they be sure that it wasn't the other way 'round -- that perceived pain affects motivation to relax -- or that some other factor really explains both pain and relaxation?"
It was Roseanne Rabbit's turn to look ashamed. "But... but.... Their findings were statistically significant!"
"Okay," said Samantha, "let's look at what they really found. How did they measure pain?"
Roseanne and Chicken Little both had to look to find the answer in the methods section. "They asked the hamsters to rate their pain on a scale of 0 to 377, with 0 being no pain and 377 being the worst pain that any hamster could imagine."
"And what kind of scores did the study subjects have on pain?"
This time, Peter had to help look, until they finally found the answer in a table they'd all ignored while reading the study text. "It says the relaxed hamsters reported a mean pain score of 2, and the tense hamsters reported a mean pain score of 1.85."
"So, how much pain is that, and how big a difference between the groups?" asked Samantha.
Chicken Little thought for a long time before she answered. "I guess this study isn't really about pain, because most of the subjects didn't have any. So the difference between the groups isn't very important."
"Right!" replied Samantha. "There's no clinical significance here."
"But how could their results be statistically significant, if they didn't mean anything?" queried Chicken Little.
"Sometimes," said Samantha, "samples can be too big for their own good. It doesn't happen often, but when it does even trivial effect sizes can become statistically significant. And Type I errors occasionally happen 9 even in the best of studies."
The sun was setting, and Peter departed to work on his newest sleep research project. Samantha offered to walk Chicken Little and Roseanne Rabbit home. Roseanne asked over and over whether Samantha thought it would be all right if she continued to teach relaxation therapies in her childbirth classes, and Samantha assured her that she shouldn't change practice on the basis of the findings of one, unreplicated study.
"But the discussion section said that was an implication of the study," said Roseanne.
"If I had a nickel for every study where the authors overstate the meaning and implications of their findings, I wouldn't need a summer internship as a forest maintenance academic to pay my tuition bills," sighed Samantha. "Roseanne, you're familiar with the theories for relief of pain. Did the authors' discussion seem logical to you?"
"Not really," admitted Roseanne, "but they claimed their findings were consistent with the Conceptual Model of Murphy 10 and that really impressed me." Roseanne reached her house and went inside, leaving Chicken Little and Samantha to finish the journey to the poultry farm alone.
"Samantha?" asked Chicken Little, "That research was really misleading. Why did it get published that way?"
"Well, Chicken Little, the forest maintenance academics do their best, but sometimes it's not enough. That's why all practitioners need to know how to evaluate the evidence for themselves. But tomorrow, you can show me what tree that study fell out of, and I'll mark it for extensive pruning or removal."
When Chicken Little arrived home, she ate a light dinner and settled herself on her roost. As she fell asleep, she vowed that she would never again be misled by study conclusions that didn't reflect the findings. And she utilized research wisely ever after.
1 At the University of Rochester School of Nursing, where unification of practice, education and research is particularly honored: University of Rochester School of Nursing (back to story).
2 p = .33 (it was spring break) (back to story)
3 Or perhaps acorns. She was in an oak grove, and it had been a long time since breakfast (back to story).
4 Unfortunately, an all-too-common but highly undesirable one (back to story).
5 As is common with research-minded canines, Douglas was part Lab by breed (back to story).
6 Douglas was also part retriever (back to story).
7 Forest maintenance interns are not issued chain saws, only lopping shears (back to story).
8 As did Waldo Wolf. See The Triads of Little Pigs for details (back to story).
9 "occasionally" = one in every twenty analyses, assuming alpha = 0.05 (back to story).
10 anything that can go wrong will, and at the worst possible moment (back to story).
© 1996 - University of Rochester School of Nursing