Power Analysis

[PsychGirl1]

I'm trying to do my first (real) power analysis and I'm running into some issues :-/. I was hoping someone here could help me out- I'd really appreciate it! I have no idea if I'm doing it correctly and my advisor hasn't given me much guidance.

I'm doing a 2x2x2 ANOVA, and I'm interested in both 2-way and 3-way interactions. So I've done a power analysis for both, and I keep getting the same n. What I'm entering in makes sense to me, but I was hoping someone could verify everything for me (thank you in advance!). Two of the factors are experimental manipulation (participants are randomized to one of four groups) and one is a measure given at baseline.

I'm using G*Power 3.1, assuming a medium effect size (since we have no idea what the effect size is), and going for 80% power with an alpha level of .05.

So for the 2-way interaction, I enter in: effect size .25, alpha .05, power 0.8, numerator df of 1 (because (2-1)(2-1) = 1), and 4 groups. Based on this, G*Power says that the critical F is 3.917, and the total sample size is 128. For the 3-way interaction, I enter in: effect size .25, alpha .05, power .8, numerator df of 1 (because (2-1)(2-1)(2-1)=1), and 8 groups. I get a critical value of 3.920 and a sample size of 128.

Am I getting the same answers because my df is 1? My advisor basically told me there should be a big difference in the number. In order to get a large increase in sample size, I basically I have to enter in up to 100 groups, and even 2 groups gives me 128. Forgive me if this is a silly question, but I'm presenting on Monday and I know my advisor will ask me why these two numbers are identical! I tried looking through power analysis resources online but I haven't found any resources that are geared to early grad students who haven't formally learned how to do power analyses yet. Since my proposal is Monday and this is a thesis, not a dissertation, please don't suggest anything fancy (like I go program a model in R or something like that), even if it would be more proper :-).

Thank you!!!!!! Again, I really appreciate any help that people can give me.

[lewin]

I've never really done a power analysis but I can tell you right now that N = 128 definitely looks too small for 8 groups because that's 16 per cell, especially for an effect size of .25. At a minimum you want n = 20/cell (e.g., as recommended recently by Simmons, Nelson, & Simonsohn, 2011).

Sorry that's not much help. Participants are cheap for my studies (undergrads or online people) so I usually just run 30-40/cell and don't bother with power analyses.

[PsychGirl1]

It's okay- thanks for your feedback!

 

I've actually been reading as many sources as possible and as far as I can tell, that number is correct for detecting two-way and three-way interactions for a 2x2x2 factorial ANOVA. Apparently one of the benefits of using a factorial ANOVA is that you can look at a lot of things without significantly increasing n (I think because you can collapse groups when comparing certain interactions). However, this power analysis only takes into account your ability to detect the interaction- it doesn't take into account the ability to decompose that interaction using simple main effects or anything like that, which would increase the n significantly. Either way, my thesis is going to be underpowered, so I'll probably just make a note of that in my presentation and hope nobody questions me too much :-). The actual study is running more people, I'm just using a subset of the sample for my thesis due to time constraints (I'm graduating in June), so there isn't much that can be done about it, and the power analysis is more of a formality.

 

But if you (or anyone else) has any feedback, suggestions, comments, or if I'm saying anything outlandishly wrong, I appreciate anything! I'm stressing out about my proposal tomorrow :-)

[yhat]

Hello!

Your power analysis seems correct (from what I can tell), however, I would be concerned about one of your a priori assumptions: the effect size. If you have no clue as to what your effect size would be for an interaction (i.e., no prior research) I would recommend a small effect size. Interactions are known for their small effect sizes. With that being said, it is often an understanding that they are hard to detect without a large sample size. One thing I would suggest is to create a range instead of a single point as your estimate. You could take an effect size of f = .10 and f = .25. For example, in G*power you can use the 'X-Y plot for a range of values' to create a graph. Here is a quick example I did:

 

 

 

 

Best.

[PsychGirl1]

Thank you!!!! This was really helpful. I really appreciate it!!