statistical comparison method?

[jojohnst]

Good day folks,

 

I've never posted on one of these forums before but would be greatly in your debt if you could assist me in completing some analysis I'm currently working on.

 

Essentially what I'm doing is comparing methods of measuring the same parameter experimentally.  I've conducted 30 experimental trials and measured the same data using 7 different well documented methods.  All of the datasets these sampling methods produced are correlated, but virtually none are the perfect 1-1 you'd hope for.

 

So really what I'm looking for are some suggestions on stats methods to help me describe the relationships between the different sampling methods and the measurement bias that is incurred based on method selection.

 

Many thanks for your time

[GeoDUDE!]

Bootstrapping might help: http://en.wikipedia.org/wiki/Bootstrapping_(statistics)

 

it might reveal some of the biases in the measuring techniques.

[Usmivka]

The best thing to do is find a statistician for help. A text targeting earth science, environmental science, or geography specific statistics, such as Modeling methods for the marine sciences (Glover, Jenkins, and Doney--an earth science focus) might also be useful. If you want to stake a stab without further consultation, I have a few ideas below.

__

 

The basic test for if two populations are the same (or two methods are both representative of the same population set of answers) is a student's t-test. Tukey's honestly significant difference test might be a good alternative. If the methods don't agree that it is the same population of experimental results, you have a problem, and you'll need to decide if one or more methods is doing a particularly poor job and why. 

 

Do all the methods have similar variable dependence? If not, you could use principle component analysis (on a single method at a time) to check where the methods are diverging and why.  For example one may weight a given component more heavily than another, with predictable divergence.

 

If you are interested in the error analysis and whether the methods are telling you the same thing within errors, then bootstrapping becomes useful, as mentioned above. Or a Monte Carlo analysis could allow you to fit a "best" answer to a given method based on the uncertainties and assuming a random distribution of error, including during your measurements--this is not always the same as the exact answer you calculate from a method, and these answers might lead to a tighter correlation.

 

Finally, sometimes the best you can do is say that the methods you believe give a certain range, mean, and mode, and the true answer lies somewhere near there. This is the unsatisfactory answer that earth scientists sometimes arrive at for parameterizations of biological, chemical, or physical rates based on some known forcing mechanism.

 

If you believe that several of the numerical methods are valid, you could combine them in a system of linear equations to constrain a solution space and optimize the "true" experimental result.

Edited September 24, 2014 by Usmivka

[jojohnst]

That is a great start - thanks a lot folks.

 

I'll dig in to all these suggestions here, also there is a group of stats folks down the hall here at my lab so I'll have some talks with them about the options too.

 

Very much appreciate your time here, points me in a good direction off the start.