Accesing Gosth Cell in FabArray<ArbitraryObject>

[FernandoIGuerra]

Dear all,

I have the current data container, and when I try to acces to the neighboors of a arbitrary box, an error in memory occurs. In the code below, when the iv_dest is called and belongs to another box, I am unable to access this position. I am using 1 ghost cell, and I cannot use the FillBoundary Function. I would be very grateful for your help.

Best regards,
Fernando

DoSplicingProcedure(
    amrex::FabArray<amrex::BaseFab<Clem::ClemManager*>>& Clem_FA,
    amrex::MultiFab& S_new,
    amrex::Real area)
{   
    Clem_FA.FillBoundary();
    for(amrex::MFIter mfi(S_new, amrex::TilingIfNotGPU()); mfi.isValid(); ++mfi){
        //const amrex::Box BxAux                              = mfi.validbox(); 
        const amrex::Box bx                                 = mfi.tilebox();
        amrex::BaseFab<Clem::ClemManager*>& clem_array      = Clem_FA[mfi];
        const amrex::Array4<amrex::Real>& snew_arr          = S_new.array(mfi);
        
        //amrex::Print() << BxAux << std::endl;
        amrex::Print()<< bx<< std::endl;
        //Think if we have to pass Boxarray to react instead of the a unique lem element
        amrex::ParallelFor(bx, [=, &clem_array] AMREX_GPU_DEVICE(int i, int j, int k) noexcept{ 

.
.
.

                  ###   **int i_dest = fluxes[dir].dir == Dir::X ? i+1 : i;
                    int j_dest = fluxes[dir].dir == Dir::Y ? j+1 : j;
                    int k_dest = fluxes[dir].dir == Dir::Z ? k+1 : k;

                  amrex::IntVect iv_dest(i_dest,j_dest,k_dest);

                    Clem::ClemManager* clem_manager_dest     = clem_array(iv_dest);

                    amrex::Vector<Clem::LemEle>& vectorSrc      = clem_manager-

                    amrex::Vector<Clem::LemEle>& vectorDes      =                                                                                 clem_manager_dest>GetVectorElement();**
                    vectorDes.insert(vectorDes.begin(), vectorSrc[0]);
                }
            }
        });
    }
}

[WeiqunZhang]

I have tried my best to fix the format. If I have made a mistake, please fix it.

[WeiqunZhang]

As you have mentioned, FillBoundary will not work. The issue is amrex::FabArray<amrex::BaseFab<Clem::ClemManager*>>'s data type is a pointer that points to memory on other processes. So you will need to find a way to communicator the data, not the data pointers. If you can tell us more details, maybe we can help.

[FernandoIGuerra]

Hello, Thank you,

ClemManager is a particle container. Then, it is a dynamic memory allocator. On deep, for each i,j,k, exists an amrex::Vector, and this vector changes its size during computation. The LemEle is where the conserved quantities are stored, emulating a particle. Digging into the code, it seems that I need to manipulate the FabArrayBase operator +() , I think FabArray doesn't let me store dynamic memory objects.

I would appreciate your help :) Thanks in advance.

Class ClemManager { private: amrex::Vector<Clem::LemEle> elements_; amrex::Real total_mass_; (More real variables)

Class LemELe{ private: std::array<amrex::Real, NUMVAR> buffer }

[WeiqunZhang]

Using your current data containers, you can pack you data into a communication buffer and use MPI to communicate the data and then unpack it. I don't think you should override opearator+ because there is not much in FabArray communication that you can reuse for your task. You also need to be careful about potential memory leaks because you are handling a lot of raw pointers. Having vectors in cells might also be very bad for performance. So you might want to reconsider your design. AMReX has particle support. Have you looked into using that?