Noise independence and betas_fithrf_GLMdenoise_RR

[johnmarktaylor91]

Hello, something of a followup on the previous issue I raised awhile back, but I am trying to understand how the procedure used by betas_fithrf_GLMdenoise_RR might affect different analysis approaches. Specifically, I am curious: does the cross-validation procedure used by betas_fithrf_GLMdenoise_RR imply that different presentations of the same stimulus are no longer truly independent? If so, does it complicate the following procedures:

1. Estimating the voxel-by-voxel noise covariance matrix. If we use the single trial betas for this (i.e., "noise" is the deviation from the stimulus-specific mean on each trial), then is it an issue that the different presentations of each stimulus aren't truly independent?

2. Using cross-validated distance metrics (i.e., crossnobis distance). Here too the assumption is that different presentations of a stimulus have independent noise and I'm wondering if betas_fithrf_GLMdenoise_RR might complicate this in some way.

If this is the case, is it more advisable to use betas_fithrf (b2) when using these analysis procedures?

To give some background (in case it's illuminating), I've been trying to use multivariate noise normalization in an ROI-identification paradigm with no success (i.e., univariate noise normalization always does better no matter what, and multivariate noise normalization often leads to very odd results), so I'm wondering if it might be related to this beta estimation procedure in some way; curious if anyone has had success with multivariate noise normalization with the NSD.

[kendrickkay]

Specifically, I am curious: does the cross-validation procedure used by betas_fithrf_GLMdenoise_RR imply that different presentations of the same stimulus are no longer truly independent?

I would say that the answer sort of depends on how stringent you are being. Technically, i would say that yes, there is some dependence, in the sense that data from multiple runs/presentations are used. But practically, i don't think there is any major/substantial dependency. More on this issue can be found here: https://glmsingle.readthedocs.io/en/latest/wiki.html#i-noticed-that-glmsingle-involves-some-internal-cross-validation-is-this-a-problem-for-decoding-style-analyses-where-we-want-to-divide-the-data-into-a-training-set-and-a-test-set

Estimating the voxel-by-voxel noise covariance matrix. If we use the single trial betas for this (i.e., "noise" is the deviation from the stimulus-specific mean on each trial), then is it an issue that the different presentations of each stimulus aren't truly independent?

For this context, I don't think there is any major issue/problem.

Using cross-validated distance metrics (i.e., crossnobis distance). Here too the assumption is that different presentations of a stimulus have independent noise and I'm wondering if betas_fithrf_GLMdenoise_RR might complicate this in some way.

I have a similar reaction (no major issue/problem) The short answer for why would be: the use of the multiple presentations is merely to set the number of nuisance regressors and the voxel-wise ridge regularization level. These are fairly coarse knobs for which it is difficult to see how substantial biases/dependences would be introduced to the multiple presentations of a given condition.

If this is the case, is it more advisable to use betas_fithrf (b2) when using these analysis procedures?

I think it's certainly interesting to compare and contrast b2 and b3 and think about the principles/issues at stake (if you are methodologically inclined). That being said, i think b3 should be fine. Also, note that what is deemed "noise" is importantly quite different across b2 and b3. (Indeed, that's the whole intention of using b3: it is removing certain aspects of the data that you probably don't want at all while still leaving "real/interesting" trial-to-trial variability in the data.)

To give some background (in case it's illuminating), I've been trying to use multivariate noise normalization in an ROI-identification paradigm with no success (i.e., univariate noise normalization always does better no matter what, and multivariate noise normalization often leads to very odd results),

Hmm. That is useful to understand the larger context. Certainly it seems like many other issues may be at play in what you are facing. The main one that immediately comes to mind is whether the procedure for estimating the structure of the multivariate noise is robust/stable enough for use in some downstream procedure. (In addition, I guess this work from us is also relevant: https://www.biorxiv.org/content/10.1101/2024.04.22.590510v1) I'd have to see in more detail what you are doing to provide a more detailed/in depth answer.

[johnmarktaylor91]

Hi Kendrick, thanks so much for this fast response and breakdown! This helps a lot with narrowing down the possibilities. It's quite possible the weird multivariate noise normalization results have nothing to do with the b2 vs. b3 differences and the issue lies elsewhere, but just to cover all the bases—is there any possibility that the use of ridge regression could be causing any unexpected complications with estimating the voxel-by-voxel noise covariance matrix? We are currently estimating this using the deviations from the stimulus-specific mean betas. I'm not sure exactly how this would manifest (e.g. maybe something to do with the voxel-specific shrinkage levels?), but just curious if anything immediately came to mind.

[kendrickkay]

"is there any possibility that the use of ridge regression could be causing any unexpected complications with estimating the voxel-by-voxel noise covariance matrix" -> Not that I can think of. I think the best and (accurate) way to think about what ridge regression is doing in this application context is that it is gently temporally smoothing adjacent trial betas (so as to reduce the impact of noise). And note that this is done completely on a per-voxel basis (hence, there are no weird voxel-to-voxel interactions that are explicitly being modified).

It seems like there is a easy (no thinking required) thing for you to try, which is to see if the weird results you are seeing also happen with b2

[johnmarktaylor91]

Indeed, got it running right now. Thanks again for your help, happy to keep you posted if we find a fix and if it'd be instructive for others in the community

[kendrickkay]

Yes, if you do discover any issues/problems/bugs, that would be good to know. (And it would be good to understand whether it falls within the domain of GLMsingle or not.)