Opened 9 years ago

Closed 9 years ago

#141 closed enhancement (fixed)

us_convert - enhancement

Reported by: demeler Owned by: dzollars
Priority: normal Milestone: future
Component: ultrascan3 Version:
Keywords: review Cc:

Description

Because us_convert is a very complex program with many steps I think it would be very helpful for the user if we had a little widget that keeps track of the various steps that need to be performed when processing data with us_convert.
I am thinking of a listbox that lists the steps, i.e.:

  1. edit run information
  2. select CP for triple 1
  3. select CP for triple 2

....

  1. select CP for triple n
  2. select solution for triple 1
  3. select solution for triple 2

....

  1. select solution for triple n
  2. Define Reference scans (if applicable, i.e., intensity data)
  3. Define Subsets (if applicable, ie. more than 2 rows in the CP)

As the user works off the various items, they disappear from the list. Not until the list is empty is "Save" activated (unless "Local disk" is selected, I guess).

The reason this came up was that originally a run had an incorrect buffer defined that was not found in the DB. I fixed this (inadvertently only for one of 8 triples) and was surprised when the error persisted. I then realized I had to click "Apply All", and then the error went away. The user wouldn't figure this out.

Change History (18)

comment:1 Changed 9 years ago by bdubbs

  • Owner changed from dan to dzollars

comment:2 Changed 9 years ago by bdubbs

  • Milestone changed from 1.0 to future

comment:3 follow-up: Changed 9 years ago by dzollars

  • Status changed from new to assigned

How can I tell when the user should define subsets?

comment:4 in reply to: ↑ 3 Changed 9 years ago by demeler

Replying to dzollars:

How can I tell when the user should define subsets?

I do not know of any way to do that part - so you will just have to leave it up to the user and maybe not include this in your test. One thing you could do is to check if the user has selected "Equilibrium" as the method in "Edit Run Info". If so, you may want to ask the question once the run info has been entered, and the user has not defined the subsets: "You selected Equilibrium data - are these data multi-channel?". You can't really tell by the shape of the data without some fancy pattern detection.

comment:5 follow-up: Changed 9 years ago by dzollars

Should the user converting RI data be prevented from saving it if he has not yet defined the reference channel and converted to pseudo-absorbance data? Currently I have the test set up in this list widget but the user is not actually prevented from saving it.

comment:6 in reply to: ↑ 5 Changed 9 years ago by demeler

Replying to dzollars:

Should the user converting RI data be prevented from saving it if he has not yet defined the reference channel and converted to pseudo-absorbance data? Currently I have the test set up in this list widget but the user is not actually prevented from saving it.

I don't see a reason right now for not converting - is the intensity profile saved after conversion? But you never know, maybe there will be a reason for not converting, so right now I would suggest to simply put out a warning message, and let the user override it if he has not converted.

comment:7 Changed 9 years ago by dzollars

  • Keywords review added

A todo list has been added as a list below the plot window. It uses the same code as the save button does to determine if there is enough information to save the data.

Also, there are some fixes and quite a few improvements in error handling, so that the user should have a much clearer understanding of what is preventing a dataset from being saved on disk or DB.

Double checked all current datatypes to verify that they still work. A lot of work on the wavelength datatype, for instance.

One thing that doesn't really work currently is knowing the difference between intensity and pseudo absorbance data once the data has been saved in the DB. So right now if you convert some intensity data, store it in the database that way, and then load it back into us_convert it will still allow you and encourage you to convert it again. Just ignore this for now. It seems to me that this is covered by ticket #161, so I am marking this ticket ready for review. Other ideas are welcome.

comment:8 follow-up: Changed 9 years ago by gegorbet

A Delete button would also be handy. Removing an experiment and AUC data cannot easily be done in us_manage_data and would be better done in us_convert. It need not remove solutions or buffers or analytes since they could be used by other experiments; but it would need to remove the entire tree of descendants from Raw on down. If it would be helpful, I could provide a utils routine for deleting all local files in a tree from the bottom up.

comment:9 in reply to: ↑ 8 ; follow-up: Changed 9 years ago by demeler

Replying to gegorbet:

A Delete button would also be handy.

s,
I think we want to be very careful with deletes. While I agree that it can be useful, especially for cleaning your disk, we need to make such deletes with the necessary warnings. Also, what I use much more in US2 is the archive functionality, where all files are compressed into a tar.gz archive which is then stored in the $HOME/ultrascan/archive directory. The question also comes up if the deletes should also occur in the database. The archive program would simply tar up the data subdirectory of results. This also brings up issues with the dependencies of rotors, buffers, solutions, analytes. When the parent experiment is archived, and then the parent is no longer available, it may be possible then to accidentally delete a solution, and when it is restored, the solution is gone. Just something to think about - should the xml entries of buffer, solution, analyte, rotor, calibrations, centerpieces, etc. be archived along with the experiment?

comment:10 in reply to: ↑ 9 Changed 9 years ago by bdubbs

Replying to demeler:

This also brings up issues with the dependencies of rotors, buffers, solutions, analytes. When the parent experiment is archived, and then the parent is no longer available, it may be possible then to accidentally delete a solution, and when it is restored, the solution is gone. Just something to think about - should the xml entries of buffer, solution, analyte, rotor, calibrations, centerpieces, etc. be archived along with the experiment?

The problem is not archiving solution, analyte, etc, but in restoring them. We have GUID's in everything. We only need to take care that we don't reproduce an element that already exists.

comment:11 follow-up: Changed 9 years ago by dzollars

In the database, deleting an experiment is relatively easy. There is a stored procedure called delete_experiment, which will delete the experiment record plus any raw data, edited data, models and noise files associated with it. It will not delete solutions or anything in that direction. I have used this procedure often in troubleshooting, and I feel good about it. So from this perspective there doesn't seem to be any reason why us_convert would do it better or more easily than another program.

From the disk, well, that's different. It would be relatively easy to delete the main directory contents, which would include the experiment, raw data, and edited data, but would not include models or noise files. While us_convert could do this, it is kind of outside the context of us_convert, if you know what I mean.

To me it makes the most sense to create a us_experiment class in utils, and add the delete capability to that. We could move or adapt other functions from us_convert's us_experiment class there too, if anybody else would use them.

comment:12 Changed 9 years ago by dzollars

I don't think we can realistically protect against everything that the disk user might do, or even a high percentage of it. As difficult as it would be to do at this point, if we're trying to do that I wonder if it might actually be less problematic to change local storage to SQLite or something. Or perhaps even MySQL on account of the stored procedures. For the effort, the end results would certainly be better.

comment:13 in reply to: ↑ 11 ; follow-up: Changed 9 years ago by gegorbet

Replying to dzollars:

In the database, deleting an experiment is relatively easy. There is a stored procedure called delete_experiment, which will delete the experiment record plus any raw data, edited data, models and noise files associated with it. It will not delete solutions or anything in that direction. I have used this procedure often in troubleshooting, and I feel good about it. So from this perspective there doesn't seem to be any reason why us_convert would do it better or more easily than another program.

The problem with a delete-experiment action in a program like us_manage_data is that it displays no experiment object. The top-level object is a Raw (AUC) object. A remove action on such an object would reasonably mean to remove the experiment, all the AUC siblings (Raw objects from the same experiment), and all descendants. But that would not be intuitive for the user who is nominally saying "remove this Raw record". Furthermore, us_manage_data knows nothing about projects, solutions, or other record types that might be (archived and) removed when an experiment is (archived and) removed. The us_convert program does have this information. A Delete or Archive button would unequivocally result in "wipe out this experiment, completely".

From the disk, well, that's different. It would be relatively easy to delete the main directory contents, which would include the experiment, raw data, and edited data, but would not include models or noise files. While us_convert could do this, it is kind of outside the context of us_convert, if you know what I mean.

To me it makes the most sense to create a us_experiment class in utils, and add the delete capability to that. We could move or adapt other functions from us_convert's us_experiment class there too, if anybody else would use them.

The us_experiment class in utils seems like a good idea. That would be a good place to engineer the local-file part of the remove, too (including looking for descendant models and noises).

comment:14 in reply to: ↑ 13 ; follow-up: Changed 9 years ago by dzollars

Replying to gegorbet:

The problem with a delete-experiment action in a program like us_manage_data is that it displays no experiment object. The top-level object is a Raw (AUC) object. A remove action on such an object would reasonably mean to remove the experiment, all the AUC siblings (Raw objects from the same experiment), and all descendants. But that would not be intuitive for the user who is nominally saying "remove this Raw record".

Is there a reason why us_manage_data doesn't start with the experiment object? That's the way I think of it.

Furthermore, us_manage_data knows nothing about projects, solutions, or other record types that might be (archived and) removed when an experiment is (archived and) removed. The us_convert program does have this information. A Delete or Archive button would unequivocally result in "wipe out this experiment, completely".

If one is deleting an experiment shouldn't the projects, solutions, etc., be retained?

comment:15 in reply to: ↑ 14 ; follow-up: Changed 9 years ago by gegorbet

Replying to dzollars:

Replying to gegorbet:

The problem with a delete-experiment action in a program like us_manage_data is that it displays no experiment object. The top-level object is a Raw (AUC) object. A remove action on such an object would reasonably mean to remove the experiment, all the AUC siblings (Raw objects from the same experiment), and all descendants. But that would not be intuitive for the user who is nominally saying "remove this Raw record".

Is there a reason why us_manage_data doesn't start with the experiment object? That's the way I think of it.

Because, already, you have Raw records that you can't do much with that mainly exist to establish the position in the hierarchy of edit records. Adding experiments would add another layer in the tree, but one you couldn't do much with. When I tried earlier to work out actions (or even information to display) on experiments, it got too complicated. Personally, I cannot reliably do things I want to do with experiments, even in us_convert.

Furthermore, us_manage_data knows nothing about projects, solutions, or other record types that might be (archived and) removed when an experiment is (archived and) removed. The us_convert program does have this information. A Delete or Archive button would unequivocally result in "wipe out this experiment, completely".

If one is deleting an experiment shouldn't the projects, solutions, etc., be retained?

Should they? The decision to retain or remove a project when deleting an experiment might be specifiable in us_convert. I don't know how you'd do that in us_manage_data. When archiving an experiment, you might want to archive projects and solutions along with it in case they got deleted elsewhere between archive and restore time. The simple parents-and-children relationship that is central to us_manage_data's simple view is not so clear when you talk about experiments and related records.

comment:16 in reply to: ↑ 15 Changed 9 years ago by demeler

Replying to gegorbet:

Replying to dzollars:

Replying to gegorbet:

<snip>

If one is deleting an experiment shouldn't the projects, solutions, etc., be retained?

Should they? The decision to retain or remove a project when deleting an experiment might be specifiable in us_convert.

Yes, they must be retained, since multiple experiments might be linked to a single project. Projects may be amended, too, as the experiment(s) progress that are associated with a project. For example, the project may be to investigate protein xyz. Now you want to do different experiments with xyz. All of this narrative describing what the experiments should do will be kept track of in the project, sort of like a lab notebook.

I don't know how you'd do that in us_manage_data. When archiving an experiment, you might want to archive projects and solutions along with it in case they got deleted elsewhere between archive and restore time. The simple parents-and-children relationship that is central to us_manage_data's simple view is not so clear when you talk about experiments and related records.

I think the safest way to do this would be as follows:

archive everything in the result/experiment directory, and then add individual rotor, calibration, project, solution, analyte, buffer, etc. xml files that contain just the experiment's details. Upon extracting the archive, the program could check the main file to see if the duplicate exists, if not, add it back into the master file. Since each record contains a GUID, it should be possible to check on disk as well. I would not worry about deleting anything from database anyway. Someone working with the database can always delete everything locally after synchronizing with the DB and restore on any other computer to get their data back, so the issue of archiving stuff from the DB is really a moot point - it is not needed - the DB is the archive.

comment:17 Changed 9 years ago by gegorbet

The talk of what can be deleted from the DB and what should always be retained, as well as talk about archiving, simply bolsters - I think - my argument that all manipulation of experiments should be done in us_convert. You cannot simply upload an experiment, for example. You must assure that any related solution, rotor calibration, project, buffer, analytes exist in the data base. Similarly, an experiment download would involve assuring that related objects exist locally.

Removing an experiment - from DB or local - must, of course, be handled very carefully. You might say that none of the peripheral objects should be deleted (since they may link to other experiments), but certainly there are cases where you do want to remove at least some of the related records; for the same reason you want to remove the experiment -- it was a test case that you have now decided is no longer useful.

Even if we greatly improve the efficiency of DB access (which I know we will do), the DB will always be slower that local disk. There will be good motivation to occasionally clean out unused DB records.

For deletions and even for upload/download, manipulation of experiments will necessarily involve bringing up special dialogs (us_solution, us_buffer, ...). The hooks to do so already exist in us_convert. It's not so straight forward in us_manage_data to bring up special dialogs and communicate user choices back to the main. Believe me, I tried; it meant trying to duplicate an awful lot of code already in us_convert.

comment:18 Changed 9 years ago by gegorbet

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.