Make FoV routine by aburrell · Pull Request #449 · SuperDARN/rst

aburrell · 2021-08-06T19:55:54Z

A preliminary version of the routine that automatically determines the backscatter return direction. This partially addresses #135.

Test results using a fitacf 3.0 file:

make_fov -cn a ~/Programs/Data/SuperDARN/Downloads/19960916.0400.00.han.fitacf3 > han.tst_fov

This creates an ascii file with 13727 lines and the following columns:

#STID DATE TIME BMNUM BMAZM CPID INTT_SC INTT_US NAVE FRANG RSEP RXRISE FREQ NOISE ATTEN CHANNEL NRANG RG GFLG FOVFLG FOV_PAST GRPFLG GRPNUM GRPID P_0 P_0_ERR V V_ERR W_L W_L_ERR P_L P_L_ERR PHI0 PHI0_ERR ELV ELV_LOW ELV_HIGH VH VH_ERR VH_METHOD REGION HOP DIST MED_P_0 MED_P_0_ERR MED_V MED_V_ERR MED_W_L MED_W_L_ERR MED_P_L MED_P_L_ERR MED_PHI0 MED_PHI0_ERR OPP_ELV OPP_ELV_LOW OPP_ELV_HIGH OPP_VH OPP_VH_ERR OPP_VH_METHOD OPP_REGION OPP_HOP OPP_DIST

Re-creating one of the figures from my paper leads to similar results. This scan is from a different time that was created during various spot checks.

Before moving this from draft status we need to:

merge Find frequency bands #444
Move general utility functions into appropriate libraries
Create a fitacf-esq non-ascii output file option
Finalize command-line options

aburrell · 2021-08-06T20:07:50Z

I am having issues compiling this on CentOS release 6.10.

Less worrying: I am getting this warning that may or may not need to be addressed:

$RSTPATH/include/base/rtypes.h:49: warning: ISO C90 does not support ‘long long’
$RSTPATH/include/base/rtypes.h:50: warning: ISO C90 does not support ‘long long’

The error (shortened for legibility):

$RSTPATH/lib/libfitacf.3.0.a(llist.o): In function `llist_sort':
llist.c:(.text+0x2d2): undefined reference to `pthread_rwlock_unlock'
llist.c:(.text+0x2e6): undefined reference to `pthread_rwlock_wrlock'
$RSTPATH/lib/libfitacf.3.0.a(llist.o): In function `llist_reverse':
llist.c:(.text+0x355): undefined reference to `pthread_rwlock_wrlock'
llist.c:(.text+0x365): undefined reference to `pthread_rwlock_unlock'
$RSTPATH/lib/libfitacf.3.0.a(llist.o): In function `llist_concat':
... et cetera ...
$RSTPATH/lib/libfitacf.3.0.a(llist.o): In function `llist_create':
llist.c:(.text+0xe26): undefined reference to `pthread_rwlockattr_setpshared'
llist.c:(.text+0xe36): undefined reference to `pthread_rwlock_init'
llist.c:(.text+0xe42): undefined reference to `pthread_rwlockattr_destroy'
collect2: ld returned 1 exit status
make: *** [make_fov] Error 1

I tried added the pthread.h and llist.h libraries to the routines that might possibly need them, but it didn't change anything.

codebase/superdarn/src.bin/tk/tool/make_fov.1.0/eval_groundscatter.c

codebase/superdarn/src.lib/tk/scan.1.7/include/scandata.h

codebase/superdarn/src.bin/tk/tool/make_grid.2.0/make_grid.c

aburrell

Also need to remove "#rawrite.c#" file.

codebase/superdarn/src.bin/tk/tool/make_fov.1.0/docs/make_fov.doc.xml

codebase/superdarn/src.lib/tk/fitmultbsid.1.0/src/write_binary.c

aburrell · 2021-10-22T15:13:59Z

Ok @ecbland and @egthomas I am a bit stuck on the last step: creating a binary file. I can write a binary without issue, but I can't read it back in. This could be:

a problem writing a valid file, or
just a reading problem.

I added a utility fitmultbsidtoascii to help test this. Would appreciate help debugging, as I am stuck and lack dmap knowledge/experience.

egthomas · 2021-10-22T15:19:54Z

@aburrell if it is a valid dmap-format file then you should be able to use

dmapdump [filename] > [output_filename]

to see all of the scalar fields in the file in plain-text format, or

dmapdump -d [filename] > [output_filename]

to see both the scalars and contents of any arrays.

aburrell · 2021-10-25T16:04:16Z

Ok @egthomas the good news is that the output file is valid! The bad news is I still have things I need to fix 🙀 But that's a better problem to have.

codebase/superdarn/src.bin/tk/tool/make_fov.1.0/test_ut_fov_struct.c

aburrell · 2021-11-23T15:35:55Z

@egthomas @ecbland this is now ready for review!

pasha-ponomarenko · 2022-02-02T23:28:22Z

@aburrell, I am a bit confused about how to test this routine.
I have installed the branch and ran make_fov on a FITACF file. The output seems to be in DMAP format rather than ASCII mentioned at the beginning of the issue. I can also see that it is a FITACF-like format but with extra fields.
Could you please provide a bit more comprehensive instructions for a non-initiated reviewer like me? :-)

aburrell · 2022-02-03T13:24:59Z

@pasha-ponomarenko before I give you detailed instructions (which I am happy to do) could you try following the examples I wrote up in the local routine docs? I think it would be good to have someone unfamiliar with the routine, but a competent RST user try to follow them so I can use your feedback to make them clearer.

But to answer a quick question you raised, there are two output formats: ascii and a dmap binary. It is basically fitacf + some fields. Not all fields are used by this routine, some are there in preparation for the rest of the backscatter ID processing that I will code up following this pull.

mts299 · 2022-02-03T14:05:00Z

@aburrell sorry what are the new fields called?

Make sure to add them to pyDARNio so people can read them :)

pasha-ponomarenko · 2022-02-03T15:35:20Z

@aburrell, do you mean this one?
https://github.com/SuperDARN/rst/blob/make_fov/codebase/superdarn/src.bin/tk/tool/make_fov.1.0/docs/make_fov.doc.xml

When I try to open it in a browser, I get this error:

It might be something very simple, but ...

aburrell · 2022-02-03T15:46:18Z

I do mean that one! Unfortunately I do not know XML, so I made the docs by copy-pasta + add sauce. Can someone with XML knowledge take a look at that file and let me know what might be wrong? @pasha-ponomarenko do you know anyone with this skill set?

@mts299 the new fields are all in the new structure housed in $RSTPATH/codebase/superdarn/src.lib/tk/fitmultbsid.1.0/. All the parameters are explained in the docs in that file, so if you don't mind trying to read them and coming back with questions, I'd appreciate your partial review!

mts299 · 2022-02-03T16:03:45Z

I will do it next week (maybe), currently cannot stare at a screen due to a concussion. Was just looking for quick things to do or ask.

pasha-ponomarenko · 2022-02-03T16:57:34Z

Apparently, there is a know issue with Github: one should never download a web file from Git as it will contain the Git webpage information as well:
process-intelligence-solutions/pm4py#247 (comment)

However, when I tried to open the same file from an RST package installed on my Linux desktop, it just showed a "raw" ASCII content and the following error(?) message:
"This XML file does not appear to have any style information associated with it. The document tree is shown below. "
This happens on both Linux and Windows.

@aburrell, are you able to see the document properly in your web browser?

aburrell · 2022-02-03T17:02:07Z

are you able to see the document properly in your web browser

No, but I don't know how to open up any of the RST xml docs. So it's a matter of me not knowing how to create something that I could open. Do you just try to open the xml files in a web browser?

pasha-ponomarenko · 2022-02-03T17:24:42Z

Yep, I believe that what you are supposed to do with XMLs.

aburrell · 2022-02-03T17:51:13Z

So, opening any of the files in a web browser doesn't do anything but display the code. I guess the good news is that my xml behaves the same as the rest of the xml. None of them open.

Given that it seems none of us have the knowledge to resolve this issue, here's what you need to run the code:

Create ascii output: make_fov -ascii
Use your own tdiff: make_fov -update-tdiff -tdiff <value_in_microseconds>
Specify your own frequency band: make_fov -tfmin -tfmax
Specify channel a: make_fov -cn a

Because you need to look at the range and azimuthal variations, you need to separate frequency bands for the analysis. For best results, use the TDIFF suited to the choses radar channel and frequency!

To convert the dmap binary to ascii, use fitmultbsidtoascii

pasha-ponomarenko · 2022-02-03T18:00:36Z

Yep, the XML business looks like a separate issue.
:-(
Thanks for the clarification. Do you have a description of the extra fields in a more robust format like DOC or PDF? Or simple ASCII?

ecbland · 2022-02-03T18:00:43Z

To view the documentation in a web browser, I think you first need to run make.doc from the root directory of RST.

cd $RSTPATH
make.doc
cd doc/html
google-chrome index.html &   # or another web browser of your choice

You can then navigate through the html documentation in the web browser.
The documentation for the make_fov binary is here: $RSTPATH/doc/html/superdarn/src.bin/tk/tool/make_fov/index.html

Added development gotchas and a summary of the FitMultBSID structures to the docs.

Added missing asterisk to the docstring.

Fixed bugs involved in loading data from multiple binary files including: - reading in multiple filenames from the command-line input, - reading in the binary header data, and - cycling through the new scans.

Made changes needed to use the new channel-dependent tdiff.

Update the parameter description for the lobe dependent elevation calculation.

Added a propagation path warning for the users.

Added the IDL tarball created when computers without IDL compile RST to the list of ignored files.

Added pointer initialization.

Added an evaluation for all elevation angles having extreme values (at the upper edge of elevation angle measurements).

Added support for extreme fov evaluation.

Updated the order in which the two fields of view are evaluated (front first) and incorporated support for extreme elevation angle evaluation.

Added a new gflg to the updated data structure. Preserves original groundscatter flag `gsct`.

Initialize a new groundscatter flag value with the original groundscatter flag value when loading scans.

Update beam evaluation to use new groundscatter flag instead of the original groundscatter flag.

Updated the groundscatter flag evaluation to alter the new flag value and maintain the old one for continuity with previous code and ease of comparison.

Some libraries that `make_fit` needs were removed during the process of resolving merge conflicts.

Fixed a typo in the structure reference.

Removed unused variables from the `make_fov` functions.

Updated the mpfit library version.

aburrell

@egthomas the compilation is failing for two routines that I can't see any connection to:

rpos_v2_test.1.0
make_sim.1.0

In both cases I think it's because the rpos.1 library is not being made correctly. Any idea why? I tried a few things without success.

aburrell · 2025-05-16T21:10:43Z

codebase/superdarn/src.lib/tk/fitmultbsid.1.0/include/fitmultbsid.h

+
+struct CellBSIDFlgs
+{
+  int gflg;


New groundscatter flag.

aburrell · 2025-05-16T21:12:37Z

codebase/superdarn/src.lib/tk/fitmultbsid.1.0/src/read_binary.c

+      if((strcmp(aptr->name, "gsct") == 0) && (aptr->type == DATASHORT)
+	 && (aptr->dim == 1))
+	{
+	  for(j = 0; j < aptr->rng[0]; j++)
+	    bm->rng[slist[j]].gsct = aptr->data.sptr[j];
+	}
+      else if((strcmp(aptr->name, "gflg") == 0) && (aptr->type == DATASHORT)
+	 && (aptr->dim == 1))
+	{
+	  for(j = 0; j < aptr->rng[0]; j++)
+	    bm->rng_flgs[slist[j]].gflg = aptr->data.sptr[j];
+	}


Now with two groundscatter flags (original and new).

aburrell · 2025-05-16T21:13:16Z

codebase/superdarn/src.lib/tk/fitmultbsid.1.0/src/write_ascii.c

+    {
+      sprintf(scan_info, "#STID");
+      sprintf(bm_info, "DATE TIME INTT_US BMNUM BMAZM CPID NAVE FRANG RSEP RXRISE FREQ NOISE ATTEN CHANNEL NRANG");
+      sprintf(rng_info, "RG GFLG NEW_GFLG FOVFLG FOV_PAST GRPFLG GRPNUM GRPID P_0 P_0_ERR V V_ERR W_L W_L_ERR P_L P_L_ERR PHI0 PHI0_ERR ELV ELV_LOW ELV_HIGH VH VH_ERR VH_METHOD REGION HOP DIST MED_P_0 MED_P_0_ERR MED_V MED_V_ERR MED_W_L MED_W_L_ERR MED_P_L MED_P_L_ERR MED_PHI0 MED_PHI0_ERR OPP_ELV OPP_ELV_LOW OPP_ELV_HIGH OPP_VH OPP_VH_ERR OPP_VH_METHOD OPP_REGION OPP_HOP OPP_DIST");


New groundscatter output is provided alongside original.

aburrell · 2025-05-16T21:14:00Z

codebase/superdarn/src.lib/tk/fitmultbsid.1.0/src/write_binary.c

+  gsct     = DataMapStoreArray(ptr, "gsct", DATASHORT, 1, &ngood, NULL);
+  gflg     = DataMapStoreArray(ptr, "gflg", DATASHORT, 1, &ngood, NULL);


New groundscatter flag provided alongside original. Original keeps the name, new flag has new name.

tables/superdarn/hdw/hdw.dat.dce

aburrell · 2025-05-16T21:22:19Z

codebase/superdarn/src.bin/tk/tool/make_fov.1.0/update_scan.c

+			  bm_new->rng[irg].elv    = bm_old.rng[irg].elv;
+
+			  /* Initialize the range gate flags */
+			  bm_new->rng_flgs[irg].gflg = bm_new->rng[irg].gsct;


New gflg is initialized here.

aburrell · 2025-05-16T21:22:54Z

codebase/superdarn/src.bin/tk/tool/make_fov.1.0/update_beam.c

+    }
+
+  /* Update the groundscatter flag */
+  EvalGroundScatter(beam);


Updates to new gflag happen here.

aburrell · 2025-05-16T21:25:34Z

codebase/superdarn/src.bin/tk/tool/make_fov.1.0/eval_groundscatter.c

+ *        - at least 3 groundscatter points are found in a 10 range gate box
+ **/
+
+void EvalGroundScatter(struct FitBSIDBeam *beam)


Updated this function to leave the original gsct flag alone.

Remove lines added during bad merge resolve.

Remove file that shouldn't be there?

…_fov

Copied file back from develop.

Removed tabs from the end of each line.

aburrell requested review from ecbland and egthomas August 6, 2021 20:09

aburrell commented Aug 30, 2021

View reviewed changes

codebase/superdarn/src.bin/tk/tool/make_fov.1.0/eval_groundscatter.c Outdated Show resolved Hide resolved

codebase/superdarn/src.lib/tk/scan.1.7/include/scandata.h Show resolved Hide resolved

aburrell linked an issue Sep 23, 2021 that may be closed by this pull request

Adding FoV detection, improved groundscatter flag #135

Open

aburrell added the enhancement label Sep 23, 2021

aburrell commented Oct 5, 2021

View reviewed changes

codebase/superdarn/src.bin/tk/tool/make_grid.2.0/make_grid.c Outdated Show resolved Hide resolved

aburrell commented Oct 12, 2021

View reviewed changes

aburrell marked this pull request as ready for review November 23, 2021 15:25

aburrell commented Nov 23, 2021

View reviewed changes

codebase/superdarn/src.bin/tk/tool/make_fov.1.0/test_ut_fov_struct.c Show resolved Hide resolved

aburrell requested a review from shibaji7 January 5, 2022 17:00

ecbland requested review from mts299 and pasha-ponomarenko January 27, 2022 16:08

ecbland added this to the RST 4.7 milestone Jan 28, 2022

aburrell and others added 17 commits May 16, 2025 15:50

DOC: added FitMultBSID documentation

ac15d87

Added development gotchas and a summary of the FitMultBSID structures to the docs.

DOC: update docstring

22c12f9

Added missing asterisk to the docstring.

BUG: fixed multi-file reading

3665177

Fixed bugs involved in loading data from multiple binary files including: - reading in multiple filenames from the command-line input, - reading in the binary header data, and - cycling through the new scans.

STY: removed sprintf string appending

5ebf791

BUG: fixed mult-file read

9ad9d73

BUG: fixes for new tdiff struct

9a6fcdd

Made changes needed to use the new channel-dependent tdiff.

DOC: update elevation docstring

6621968

Update the parameter description for the lobe dependent elevation calculation.

DOC: added prop warning

ac50601

Added a propagation path warning for the users.

ENH: added idl.tar.gz to .gitignore

f54579d

Added the IDL tarball created when computers without IDL compile RST to the list of ignored files.

BUG: init pointer

cdc02b5

Added pointer initialization.

ENH: added extreme value analysis

5f0a765

Added an evaluation for all elevation angles having extreme values (at the upper edge of elevation angle measurements).

ENH: added extreme fov support

6558afe

Added support for extreme fov evaluation.

ENH: updated extreme fov and fov order

11313e6

Updated the order in which the two fields of view are evaluated (front first) and incorporated support for extreme elevation angle evaluation.

ENH: added new gflg

e738cc9

Added a new gflg to the updated data structure. Preserves original groundscatter flag `gsct`.

ENH: initialize new gflg value

9a08609

Initialize a new groundscatter flag value with the original groundscatter flag value when loading scans.

ENH: update beam evaluation

9418072

Update beam evaluation to use new groundscatter flag instead of the original groundscatter flag.

ENH: updated groundscatter evaluation

65d0dad

Updated the groundscatter flag evaluation to alter the new flag value and maintain the old one for continuity with previous code and ease of comparison.

aburrell force-pushed the make_fov branch from d72971c to 65d0dad Compare May 16, 2025 19:51

aburrell added 4 commits May 16, 2025 16:26

BUG: re-add make_fit libraries

45e0b15

Some libraries that `make_fit` needs were removed during the process of resolving merge conflicts.

BUG: fixed typo in structure

2e0ee47

Fixed a typo in the structure reference.

BUG: removed unused variables

18d988e

Removed unused variables from the `make_fov` functions.

BUG: updated library version

fec5a20

Updated the mpfit library version.

aburrell commented May 16, 2025

View reviewed changes

aburrell and others added 7 commits May 16, 2025 17:29

DOC: Update tables/superdarn/hdw/hdw.dat.dce

b7f9cdc

Remove lines added during bad merge resolve.

REV: remove odd file

f0396ec

Remove file that shouldn't be there?

Merge branch 'make_fov' of https://github.com/SuperDARN/rst into make…

64c7d0c

…_fov

REV: file should be there

466a191

Copied file back from develop.

STY: removed tabs

aec1aed

Removed tabs from the end of each line.

Merge branch 'develop' into make_fov

ba668fd

Fixing compilation errors in make_fov branch

661198c

		gsct = DataMapStoreArray(ptr, "gsct", DATASHORT, 1, &ngood, NULL);
		gflg = DataMapStoreArray(ptr, "gflg", DATASHORT, 1, &ngood, NULL);

Comments

Conversation

aburrell commented Aug 6, 2021 • edited by pasha-ponomarenko Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aburrell commented Aug 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aburrell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aburrell commented Oct 22, 2021

Uh oh!

egthomas commented Oct 22, 2021

Uh oh!

aburrell commented Oct 25, 2021

Uh oh!

Uh oh!

aburrell commented Nov 23, 2021

Uh oh!

pasha-ponomarenko commented Feb 2, 2022

Uh oh!

aburrell commented Feb 3, 2022

Uh oh!

mts299 commented Feb 3, 2022

Uh oh!

pasha-ponomarenko commented Feb 3, 2022

Uh oh!

aburrell commented Feb 3, 2022

Uh oh!

mts299 commented Feb 3, 2022

Uh oh!

pasha-ponomarenko commented Feb 3, 2022

Uh oh!

aburrell commented Feb 3, 2022

Uh oh!

pasha-ponomarenko commented Feb 3, 2022

Uh oh!

aburrell commented Feb 3, 2022

Uh oh!

pasha-ponomarenko commented Feb 3, 2022

Uh oh!

ecbland commented Feb 3, 2022

Uh oh!

aburrell left a comment

Choose a reason for hiding this comment

Uh oh!

aburrell May 16, 2025

Choose a reason for hiding this comment

Uh oh!

aburrell May 16, 2025

Choose a reason for hiding this comment

Uh oh!

aburrell May 16, 2025

Choose a reason for hiding this comment

Uh oh!

aburrell May 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aburrell May 16, 2025

Choose a reason for hiding this comment

Uh oh!

aburrell May 16, 2025

Choose a reason for hiding this comment

Uh oh!

aburrell May 16, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

aburrell commented Aug 6, 2021 •

edited by pasha-ponomarenko

Loading

aburrell commented Aug 6, 2021 •

edited

Loading