proposed workflow for working with partial sample-level records via web UI

[ksg52]

@wernersun @vkuznet @jpk96 @suchismitasarker @esfyee @adas42

Summarizing from a discussion yesterday -

On the FOXDEN dev instance, Valentin has implemented an option to create a "template" record: https://foxden-dev.classe.cornell.edu:8344/tmpl/records. Right now the examples there use the parts of the 3A schema that we currently have users provide on a sample-by-sample basis. The idea is to provide a web UI where users can create temporary, partial records with their sample information prior to data collection. Partial records can be copied and then edited, different versions saved, etc... this addresses the pain point of having to re-enter the same information repeatedly for samples that are similar to one another (e.g. same alloy with different heat treatment, etc)

We came up with a proposed workflow for merging these partial records with BTR-level information and information best populated directly from SPEC at the time of data collection:

1. SPEC generates partial BTR-level record (probably via beamline scientist input - this is how 3A currently does it)
2. SPEC provides a URL for accessing the web UI for partial, sample-level template records, showing all the templates associated with BTR via this FOXDEN end-point /tmpl/records?btr=btr-id3

• This URL should also be available before beamtime, so beamline scientists can point users to the web UI.

3. User uses this web UI to create partial records from template
   • Note: FOXDEN will ensure that template label is not empty string and should be unique
4. When it's time to start data collection on a specific sample, user goes to SPEC and calls newsample SPEC macor, "newfile", or whatever the relevant station macro is, along with the label name of the specific sample's partial record
5. SPEC fetches the partial record corresponding to the provided BTE and label name pair, e.g. /tmpl/records?btr=btr-id3a&label=test-label23
6. SPEC macro provides functionality to merge partial user record with the BTR-level record from (1), plus any sample-level information that SPEC is better suited to populate
   • at 3A, this would be things like the raw data location
7. SPEC submits the final record to FOXDEN.
   Valentin requested that the scientists tagged here reach a consensus on whether this is how we want to proceed in the next ~2 weeks - can folks please respond here? This is also a good time/place to raise concerns or things that need clarification.

[wernersun]

Thanks @ksg52 for this very clear proposal. A few comments and questions:

• Thinking about the workflow from users' POV, their home base is moving from the SPEC terminal to the FOXDEN web interface (menu of templates). I.e. they should not even start a new dataset in SPEC without thinking about metadata first. I think this is a desired behavior to be reinforced.
• However, this also introduces more cognitive load for context switching from FOXDEN to SPEC. Is this a concern?
• Going from steps 3 to 4: I worry about transcription errors in the template name. How can we make this transition foolproof?
• Which features should scientists be able to enable/disable?
• In step 7, I suggest implementing asynchronous injection like at 1A3: SPEC writes json files into a staging area on disk instead directly into FOXDEN. Then, there is a service that periodically injects the json files in the staging area into FOXDEN. This way, if something goes wrong with FOXDEN, it doesn't block data collection and metadata is still recorded.
• Merging 3 fragments into a complete metadata record (btr-level, spec-automated, user-provided): this functionality seems more natural to implement in FOXDEN, not SPEC (it's a variation on the existing amend button) and can be incorporated into the asynchronous process in the previous bullet.

[suchismitasarker]

Thanks @ksg52 for the summarizing the details!

Below are my comments for 4B regarding the proposed workflow for merging partial records with BTR-level information and the metadata that is best populated directly from SPEC at the time of data collection.

Currently, SPEC generates a partial BTR-level record for 4B using the newbtr macro. At SPEC, my sample information metadata is already tagged within my experimental data collection workflow, so users must provide that information before collecting data. Once data collection is finished, SPEC submits the final record directly to FOXDEN.

I like the idea of partial sample information gathering at the URL when accessing the web UI. At the moment, I collect that information in a Google document, so having it integrated into FOXDEN would be very helpful for 4B users. Ultimately, if FOXDEN can pull the correct information automatically, it will be extremely beneficial, as users are currently providing the same information twice—once in the Google doc and again in SPEC.

@vkuznet I need to help for the Web UI for 4B.

[ksg52]

To a few of @wernersun 's comments:

* Thinking about the workflow from users' POV, their home base is moving from the SPEC terminal to the FOXDEN web interface (menu of templates). I.e. they should not even start a new dataset in SPEC without thinking about metadata first. I think this is a desired behavior to be reinforced.

Agreed.

* However, this also introduces more cognitive load for context switching from FOXDEN to SPEC. Is this a concern?

In my opinion, no, at least not we do it right. At 3A, users already experience a significant context-switching burden to set up their data collection - many of our standard scan strategies require them to create input files in either a text editor or via a Python script, and then point SPEC to those files to execute data collection. The proposed metadata record strategy isn't too different from that and has the benefit of letting users front-load some of the work by prepping and staging sample metadata in advance.

* Which features should scientists be able to enable/disable?

The ability to fully disable interaction between SPEC and FOXDEN services is important, to accommodate cases where a group doesn't want to use FOXDEN or unforseen errors/conflicts hamper data collection and we need to triage.

* In step 7, I suggest implementing asynchronous injection like at 1A3: SPEC writes json files into a staging area on disk instead directly into FOXDEN. Then, there is a service that periodically injects the json files in the staging area into FOXDEN. This way, if something goes wrong with FOXDEN, it doesn't block data collection and metadata is still recorded.

Strongly agree.

* Merging 3 fragments into a complete metadata record (btr-level, spec-automated, user-provided): this functionality seems more natural to implement in FOXDEN, not SPEC (it's a variation on the existing amend button) and can be incorporated into the asynchronous process in the previous bullet.

Agreed.

[vkuznet]

@ksg52 I would like to understand if we settle on these requirements and will move forward with them.

@keara-soloway , I have the FOXDEN API to look-up template records for given BTR or BTR/label pair. Here is concrete examples:

# fetch template records for given BTR to address step #2
https://host:port/tmpl/records?btr=btr-id3a

# fetch template record for given btr and label to address step #5
https://host:port/tmpl/records?btr=btr-id3a&label=test-label23

The above calls will return list of template records matched the given input parameters. The actual HTTP request will need a token and appropriate HTTP header to specify MIME type, e.g. here is how actual curl call looks like to my local server:

curl -H "Authorization: bearer $t" -H "Accept: application/json" \
     "http://localhost:8344/tmpl/records?btr=btr-id3a&label=test-label23"

Could you please incorporate them into SPEC macros. You'll need to replace host and port with associative foxden production/test instances.

Finally, we will need to implement in SPEC merge action (this can be optional macro which beamlines may choose to use) from record received from FOXDEN and SPEC auto-generated record.

[ksg52]

@adas42 @esfyee @jpk96 can you weigh in on if you're on board with the proposed workflow? Are there things you'd like to have addressed before Valetin moves ahead?

[adas42]

I am onboard. One question: some of our users have sample level metadata record jsons created, for a few standard samples. Will those become unusable?

[vkuznet]

@adas42 we can incorporate user metadata as well but it will be just a blob, i.e. a schema will not care of its data-type and structure. In the web UX users will fill it out as they like, i.e. put JSON or list of numbers or anything else. I'll see what can be done in web interface.

[jpk96]

@ksg52 I agree with the proposed workflow. For the going from steps 4 to 5, I can imagine a few implementation details that would make the process more robust.

• If the specified sample partial record (identified by name or DID) cannot be found, display an error message prompting the user either to provide a different name or to proceed anyway. If the user chooses to proceed, the final record would be created without the sample record, and they could later supplement the SPEC-submitted record with the missing sample record.
• The SPEC data file could include a comment indicating either the name/DID of the referenced sample partial record or that the specified partial record could not be found.
• I could also imagine this workflow being incorporated into a user-defined macro. In that case, it would be important to add an optional argument to the newsample command that explicitly instructs the workflow to continue even if the specified sample partial record does not exist. This would allow automated workflows to proceed without requiring interactive user input. Again, the SPEC data file could include a comment indicating either the name/DID of the referenced sample partial record or that the specified partial record could not be found.

[keara-soloway]

For step 4, should newsample now be supporting an additional argument -- the label of the desired template record? Should it be possible to use a values.json AND a template record label with newsample? If so, and if both are used, should any duplicated values from the web template record override those in values.json, or the other way around?

[vkuznet]

For step 4, should newsample now be supporting an additional argument -- the label of the desired template record? Should it be possible to use a values.json AND a template record label with newsample? If so, and if both are used, should any duplicated values from the web template record override those in values.json, or the other way around?

Keara, those are good questions. The answer is no, since template record is not persistent and can be modified by end-user. The workflow should be:

• SPEC generates values JSON
• user redirected to FOXDEN and modify his/her template record
• user return to SPEC and provide label for newsample
• SPEC fetches the template record for given template.

If both (newsample and label) are used and if SPEC fetches ONE template record corresponding to that label then template record overrides those values in JSON. I would expect something like spec_foxden_record.update(template_record) (in python syntax) or something similar in SPEC.

If it will not be still clear we may discuss the flow in zoom/Teams call.

[keara-soloway]

After discussing with Valentin over a call:

newsample DOES need to support a new optional argument for template record label. We should support any of the following usages for newsample:

# Non-automated sample-level fields are filled out (or not) according to mode selected in foxden_setup
newsample sample_name

# Non-automated sample-level fields are taken from values.json
newsample sample_name values.json

# Non-automated sample-level fields are taken from the unique template record identified by "https://<foxden_url>/tmpl/records?btr=<current_btr>&label=template_label"
newsample sample_name template_label

# Non-automated sample-level fields are taken from values.json AND the unique template record identified by "https://<foxden_url>/tmpl/records?btr=<current_btr>&label=template_label"
newsample sample_name values.json template_label
newsample sample_name template_label values.json

For the last two cases, can scientists weigh in on which values should take precedence if there are any duplicate fields between values.json and the labelled template record?