ACQ4 is a data acquistion program written in Python (see www.acq4.org and Campagnola et al., Front. Neuroinformatics, 2014). ACQ4 supports multiple acquistion modes, cameras, lasers, etc, and remains in active development as a platform for neurophysiology.
"NWB" is a common file format for neurophysiology data (see Neurodata without borders).
The conversion uses pynwb (https://pynwb.readthedocs.io/en/stable/index.html).
Note 1: Two kinds of optical stimulation are supported in the current version of this translator. One is "wide field LED" illumination, where the location of the stimulus is determined by the objective location relative to the cell (assuming epi-illumination). The other is for laser-scanning photostimulation, where the laser spot is guided by x and y steering mirrors and flashed to stimulate neural elements (or uncage photolabile compounds). In this case, the scanning spot positions are captured as well
Note 2: The resulting data files may be quite large. We do not include he slice and cell images in this conversion process as yet. These are can be generated by the Mosaic Editor in acq4, including fiduciary points, cell locations, and mixes of brightfield and fluorescence images. The images created by the Mosaic Editor (which may involve a processing pipeline from the raw data) can be stored alongside the original data. Usually these are on a per-slice basis.
The 'acq4_to_nwb.py' program should be run under the environment created for the ephys package. I usually do this by going to the ephys source directory, and activating the ephys environment (if it needs to be updated or built, run './make_env.sh' first, then 'ephys_venv/bin/activate' (on Unix-like systems)). Then, jump back to the Dandi directory to run the code. The ephys package is required because we need a number of tools, including the acq4_reader, to read the acq4 data. acq4_reader is a part of ephys, and there is no stand-alone reader at this time.
Modify acq4_to_NWB.py so that the main() function picks up the database for the data to be converted. The database is generated by the dataSummary command line tool in ephys. This script scans directories of acq4 data, and generates a pickled Pandas file with the metadata, and from that generates an excel file that is human readable. The dataSummary tool can handle subdirectories (in the case that the data is broken up by recording condition, slice angle, etc).
The datasummary file needs to be known and readable in acq4_to_NWB.
The output files are stored in the "test_data" directory, with a compact filename for each cell. The format is:
ExperimentName~Rig#~year.mo.da~SnnCmm.nwb
where: Snn refers to the slice number for the day, and Cmm refers to the cell number within that slice.
Once you have generated the dataSummary file and have edited the program to point to the right directories, run:
python src/acq4_to_nwb.py
This may take some time.
See: Using Dandi
Once all files have been converted it is time to inspect the data structure to be sure that it is acceptable for NWB readers and for the Dandi archive:
nwbinspector <source_folder> --config dandi
or:
nwbinspector <source_folder> --config dandi --report-file-path <report_location>.txt
If there are errors that are "CRITICAL", you should fix them in the original data (the acq4 files), then re-run acq4_to_nwb.py to generate new files. Other errors you may need to fix are genotypes, and other missing metadata.
Next validate using dandi:
dandi validate <source_folder>
If you have a dataset and plan to append to it or to update it, first download the current version:
dandi download https://dandiarchive.org/dandiset/<dataset_id>/draft
This may give "FileExistsErrors". It is mainly (I think) so that the files currently in a dataset include the previous (already uploaded) files.
Now:
cd <dataset_id>
dandi organize <source_folder> -f dry
dandi organize <source_folder>
dandi validate .
For final:
dandi upload
or for staging:
dandi upload -i dandi-staging
Use display_nwb.py to view the data in the nwb files. This little program plots the data for each protocol to a new matplotlib window. The upper left plot has the individual traces, stacked, and the plot beneath this (middle left) has the command (current or voltage) superimposed. Note that the order of the traces in the upper left is out of sequence. Protocols with only current-clamp or voltage-clamp data and no optogenetic stimulation will have an empty upper right (stimulus sites) and lower left (command voltage to LED or laser) plots. The lower right plot is the average voltage or current (primary recording), while the plot above this is from one trace (usually trace 0). When laser scanning maps are displayed, this trace should be able to be changed by clicking on the map points in the upper right trace. When LED data is displayed, only the first trace is shown. Units are in volts and amperes. The top title shows the name of the file, and the name of the protocol.
V0.1.0 6/23/2023 First version. Much data held in 2D arrays (no longer allowable). V0.2.0 6/21/2024 Second version. Reorganized NWB format, read more opto stimulation information.
Last updated 6/26/2024 Paul B. Manis