Segfault in brain::neuron::Soma::getProfilePoints

[mgeplf]

Backtrace is:

60      /nvme//spack-stage-brion-3.3.0-uhz33z2f5zwooo35gzjg54hwc7givx5x/spack-src/brain/neuron/soma.cpp: No such file or directory.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7_6.6.x86_64 libicu-50.1.2-17.el7.x86_64
(gdb) bt
#0  brain::neuron::Soma::getProfilePoints (this=this@entry=0x7fffffffcd00) at /nvme/spack-stage-brion-3.3.0-uhz33z2f5zwooo35gzjg54hwc7givx5x/spack-src/brain/neuron/soma.cpp:60
#1  0x00007fffed48ef23 in brain::neuron::Soma::getMeanRadius (this=this@entry=0x7fffffffcd00) at /nvme//spack-stage-brion-3.3.0-uhz33z2f5zwooo35gzjg54hwc7givx5x/spack-src/brain/neuron/soma.cpp:65
#2  0x00007fffed6cf19c in ems::EventsLoader::_computeStaticEventGeometry(std::vector<std::tuple<unsigned long, unsigned int, unsigned int, unsigned short>, std::allocator<std::tuple<unsigned long, unsigned int, unsigned int, unsigned short> > > const&, std::vector<std::shared_ptr<brain::neuron::Morphology>, std::allocator<std::shared_ptr<brain::neuron::Morphology> > > const&) () at ../emSim/EventsLoader.cpp:116
#3  0x00007fffed6d0671 in ems::EventsLoader::_loadStaticEventGeometry() () at ../emSim/EventsLoader.cpp:87
#4  0x00007fffed6d0c42 in ems::EventsLoader::EventsLoader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, glm::vec<2, float, (glm::qualifier)0> const&, float) () at ../emSim/EventsLoader.cpp:44
#5  0x00000000004160fe in process(EmsimParams const&) () at ../apps/emsimLFP/main.cpp:131
#6  0x0000000000415d55 in main () at ../apps/emsimLFP/main.cpp:176
#7  0x00007fffec447495 in __libc_start_main () from /lib64/libc.so.6
#8  0x0000000000415e8f in _start () at ../apps/emsimLFP/main.cpp:181

This is with the MOOC circuit (proj112/simulations/CA1.O1.mooc-circuit/LFP/full_dt05//BlueConfig); a couple things that may help with debug:

1. This uses SWC files (as determined by the BlueConfig MorphologyType swc), however, I see it try and access the h5 morphs first:

stat("CA1.O1/mooc-circuit/morphologies/swc//011127HP1_-_Scale_x1.000_y0.850_z1.000.h5", 0x7fffffffc730) = -1 ENOENT (No such file or directory)
stat("CA1.O1/mooc-circuit/morphologies/swc//011127HP1_-_Scale_x1.000_y0.850_z1.000.swc", {st_mode=S_IFREG|0664, st_size=3210557, ...}) = 0

2. There is an warning:

[Brion][Warning]Requested 18198 GIDs [1:18198] are not a subset of the 18198 GIDs in the report [0:18197[Brion][Warning]], using intersection size 18197 [1:18197]

So I'm wondering if it's a mismatch in GIDs; it appears that the following SONATA files are being loaded:

proj42/circuits/CA1.O1/mooc-circuit/sonata/nodes/nodes.h5
proj42/circuits/CA1.O1/mooc-circuit/sonata/edges/edges.h5
proj112/simulations/CA1.O1.mooc-circuit/LFP/full_dt05//AllCompartmentsIMembrane.h5

Any ideas?

[NadirRoGue]

For 1. Brion tries first to find H5 morphologies, if it fails, it will proceed with SWC:

    URI getMorphologyURI(const std::string& name) const
    {
        if (boost::filesystem::path(name).is_absolute())
            return URI(name);

        URI uri(getMorphologySource());
        const auto h5 = uri.getPath() + "/" + name + ".h5";
        if (!fs::exists(fs::path(h5)))
        {
            const auto swc = uri.getPath() + "/" + name + ".swc";
            if (fs::exists(fs::path(swc)))
            {
                uri.setPath(swc);
                return uri;
            }
        }
        uri.setPath(h5);
        return uri;
    }

For 2., I see that the circuit defines the node ids starting at 0:

nroman@bbpv1:~$ h5dump -d /nodes/hippocampus_neurons/node_type_id /gpfs/bbp.cscs.ch/project/proj42/circuits/CA1.O1/mooc-circuit/sonata/nodes/nodes.h5
HDF5 "/gpfs/bbp.cscs.ch/project/proj42/circuits/CA1.O1/mooc-circuit/sonata/nodes/nodes.h5" {
DATASET "/nodes/hippocampus_neurons/node_type_id" {
   DATATYPE  H5T_STD_I64LE
   DATASPACE  SIMPLE { ( 18198 ) / ( 18198 ) }
   DATA {
   (0): -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
   (17): -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
   (34): -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
   (51): -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
   (68): -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,

Which is consistent with what can be found in the report:

nroman@bbpv1:~$ h5dump -d /report/hippocampus_neurons/mapping/node_ids /gpfs/bbp.cscs.ch/project/proj112/simulations/CA1.O1.mooc-circuit/LFP/full_dt05//AllCompartmentsIMembrane.h5 > report.txt
nroman@bbpv1:~$ cat report.txt | grep " 0,"
   (654): 17205, 17242, 17297, 17339, 17344, 0, 932, 960, 1896, 2323, 3122,

However, Brion get the circuit GIDs increasing by 1 their index in the circuit, to make them always start at 1 (I thought myself aswell that GIDs started at 1 and not at 0):

    GIDSet getGIDs() const
    {
        brain::GIDSet gids;
        brain::GIDSet::const_iterator hint = gids.begin();
        for (uint32_t i = 0; i < getNumNeurons(); ++i)
            hint = gids.insert(hint, i + 1);
        return gids;
    }

As for the SEGFault, I have sucessfully loaded the circuit /proj112/simulations/CA1.O1.mooc-circuit/LFP/full_dt05/BlueConfig with Brion, loaded the morphologies and printed all the mean_radius of all the somas. I am not sure if it could be a problem of the GIDs which EMSim is feeding Brion, or something else.

[mgeplf]

Thanks for looking, so 1 & 2 are red-herrings.

As for the SEGFault, I have sucessfully loaded the circuit /proj112/simulations/CA1.O1.mooc-circuit/LFP/full_dt05/BlueConfig

Ok, that's good to know - narrows the problem down a lot.

Have you guys had to make many changes to the apps that use Brion to load? Does the above problem strike you as something you've seen before? emSim as originally done by the viz team, so I don't know the code very well.

[NadirRoGue]

It's the first time I see this kind of error. My guess is that, because GIDs start at 0, but Brion load them starting at 1, it might be trying to load an unexisting morphology.

We ask MVDTool for the morphology name of a given GID set to build the path to their morphology files. It might be asking MVDTool for a GID that does not exists, 18198, returning an empty Morphology name. This would cause the underlying object used by neuron::Morphology to be null, because it could not be loaded, and would explain the place where the segmentation fault is happening (which is exactly when this underlying object is called)

[mgeplf]

The suspicious call site is here https://github.com/BlueBrain/EMSim/blob/master/emSim/EventsLoader.cpp#L81:

gidsPerBatch contains { 1 2 3 .... 1000 }

And the dereference of morphologies at index 0 happens at https://github.com/BlueBrain/EMSim/blob/master/emSim/EventsLoader.cpp#L112

Is there a way that the return value of loadMorphologies() is getting invalidated, or hasn't finished yet?

[NadirRoGue]

The value of loadMorphologies() is valid at the time of returns. Each of the returned object, has an inner object that is loaded asynchronously. If the inner object is accessed while the load hasnt finish, it will block until its done. The inner object might fail to load, but that should trigger an exception, since the morphology is ulitimately loaded by MorphIO, which checks for the validity of the morphology file.

The loading/call trace:

brain::neuron::Morphology
   -> brain::neuron::Morphology::Impl
           -> brion::Morphology
                   -> brion::Morphology::Impl
                          -> brion::plugin::MorphologyMORPHIO