Opened 9 years ago

Closed 9 years ago

#209 closed defect (fixed)

backend 2DSA is going way too slow

Reported by: demeler Owned by: gegorbet
Priority: critical Milestone: future
Component: ultrascan3 Version:
Keywords: review Cc:

Description (last modified by gegorbet)

The supercomputer version of 2DSA runs significantly slower than the desktop version on my slow 5 year old laptop - something is wrong, but I have no idea what.

Review-ready.

Attachments (1)

performance-comparison-us2--us3.ods (9.3 KB) - added by demeler 9 years ago.

Download all attachments as: .zip

Change History (16)

comment:1 Changed 9 years ago by bdubbs

  • Status changed from new to assigned

comment:2 follow-up: Changed 9 years ago by bdubbs

Gary and I went over this and found three errors.

  1. alamo was only using 8 processors instead of 32.
  2. The noise table was not being updated properly. Wrong model ID in some cases, but right model GUID.
  3. On all passes except the first, ready processors were being locked out, effectively using only a single worker.

The first problem was in lims and has been fixed for all instances. The second was in the global cleanup routine. The third was/is in the back end analysis code. The back end has not been committed to svn yet and only fixed on alamo. Please give alamo a try. If it works OK I'll update svn and all the other clusters.

comment:3 in reply to: ↑ 2 Changed 9 years ago by demeler

Replying to bdubbs:

Gary and I went over this and found three errors.

  1. alamo was only using 8 processors instead of 32.
  2. The noise table was not being updated properly. Wrong model ID in some cases, but right model GUID.
  3. On all passes except the first, ready processors were being locked out, effectively using only a single worker.

The first problem was in lims and has been fixed for all instances. The second was in the global cleanup routine. The third was/is in the back end analysis code. The back end has not been committed to svn yet and only fixed on alamo. Please give alamo a try. If it works OK I'll update svn and all the other clusters.

The speed problem seems to be resolved, it works much quicker now. Although even 8 cores should have been a lot faster than my old 2 cores on my laptop. But point (3) explains why the 8 procs were not making a difference. However, the lims update still failed with this MySQL error message:

UPDATE noise SET editedDataID=(SELECT editedDataID FROM editedData WHERE editGUID='1db58798-4a9d-4ab3-b5b2-eb0e1818231c'),modelID=(SELECT m.modelID FROM model m, noise n WHERE m.modelGUID=n.modelGUID AND noiseID=94)WHERE noiseID=94
You can't specify target table 'noise' for update in FROM clause

Let me know when this is fixed and I will try again.

comment:4 Changed 9 years ago by bdubbs

  • Keywords review added

Fixed the cleanup.php sql error.

comment:5 Changed 9 years ago by dzollars

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:6 Changed 9 years ago by demeler

  • Resolution fixed deleted
  • Status changed from closed to reopened

I am opening this ticket again. While the speed has improved, I have now checked it with the same number of scans and similar grid settings against us2 and there is still a significant speed difference:

  1. triples 2/4/6B230 were analyzed with similar editing settings
  2. analysis included ti noise correction, but no meniscus fitting
  3. analysis settings were 16 grid repetitions, resolution 10 for both s and f/f0.

The results are attached in a spreadsheet.

Changed 9 years ago by demeler

comment:7 follow-up: Changed 9 years ago by demeler

  • Status changed from reopened to new

The performance comparison spreadsheed shows that there is still a significant difference between the backend reported RMSD and the RMSD observed in fematch. This would suggest that there may still be differences in hydrodynamic parameters. This should be printed out and verified to be the same.

comment:8 in reply to: ↑ 7 Changed 9 years ago by bdubbs

Replying to demeler:

The performance comparison spreadsheed shows that there is still a significant difference between the backend reported RMSD and the RMSD observed in fematch. This would suggest that there may still be differences in hydrodynamic parameters. This should be printed out and verified to be the same.

I can add input or calculated parameters to the email sent. Which would you suggest?

comment:9 Changed 9 years ago by bdubbs

  • Keywords review removed
  • Milestone changed from beta to future
  • Status changed from new to assigned

comment:10 Changed 9 years ago by dzollars

  • Priority changed from normal to critical

comment:11 Changed 9 years ago by bdubbs

See comments in #217 also.

comment:12 Changed 9 years ago by gegorbet

  • Description modified (diff)
  • Keywords review added
  • Owner changed from bdubbs to gegorbet
  • Status changed from assigned to new

Changes made - thus far, only on BCF - not only seem to fix the discrepancies in FE/BE results, but also add - via a utils version of calc_residuals() - significant speedups in computation time. For 2DSA noise calculation, 35% was shaved off of worker run-time.

comment:13 Changed 9 years ago by gegorbet

  • Status changed from new to assigned

comment:14 Changed 9 years ago by gegorbet

Borries will be running jobs to verify that the speed issue is resolved.

comment:15 Changed 9 years ago by dzollars

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.