From nch@roe.ac.uk Fri Mar 2 16:46:45 2007 Date: Fri, 2 Mar 2007 16:35:17 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project team meeting mins, 2/3/07 Minutes of WFAU VDFS Science Archive meeting: 2nd March 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, ETWS, RSC, LGR, NJC, MAR, MSH, JB, JPE Apologies: JMS, JDT, BCW, AL DONM: 10am, Friday 9th March 2007 in the Plate Library Actions discharged this week: ----------------------------- None this week. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues; GLIMPSE done; 2MASS and BestDR2 remain to be done (latter has been shrunk in preparation, recovering around 200GB of space). ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete. o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; NCH, RSC and MAR discussed this in the week and because of some subtle complexities it will be put on hold until after DR2. RSC has made some code changes to make the curation applications robust against this very minor schema change when it does happen. ACTION: NCH and RSC to refactor CU19 next week. Continues; NCH & RSC have had an initial look and with a view to scalability for the GPS (and also VISTA in future) a couple of experiments comparing large table copying options are being tested prior to implementation in the production code. Actions carried forward from 23/02/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. Specific points and new actions: -------------------------------- Project management: JPE noted that there is a VDMT next week, and that there is the outstanding issue of the next VDUC. ACTION: JPE to contact Tim Naylor concerning scheduling a VDUC meeting. PMW noted that the progress reports for next week's VDMT will be brought around for filling in offline on Monday. ACTION: PMW to prepare the usual VDMT materials early next week. WFCAM & VISTA updates: JPE gave an update on Vista. M2 is to be delivered within the next month; M1 is on schedule for delivery in June. VirCAM has been tested for mechanical interfacing with the telescope and all is well; cold tests of the camera were delayed slightly by plumbing problems. VirCAM on-sky is anticipated in September. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 21st Feb. ETWS noted that new WFCAM filters are being used; NJC volunteered to find out the details and insert them into the ingest DB. ACTION: NJC to communicate with JAC over the new filters being used in WFCAM, and input the necessary data into the WSA. The team noted with appreciation the efforts being made at the CASU end to push forward UKLight things with UKERNA. Networking: NCH reported receipt of an 06B processing update from MR at CASU (thanks!); this has been forwarded to JB and ETWS. Sorting out the data is obviously going to take no small amount of hard work at CASU and WFAU very much appreciate the efforts being made. MJI is checking over the first small batch of processed data prior to release to WFAU; NCH noted that we must endeavour to get the data copied up, JPEGed and ingested asap to test for any problems and also to allow some external (UKIDSS) checks for rapid feedback to WFAU/CASU in order to pick up any problems in good time. WSA Operations: NCH noted that releasing the GPS is now the priority, but that normal transfer and ingest operations must go ahead in parallel for the fresh 06B data (thanks to ETWS for refactoring the early CU software to facilitate this). JB noted: "WORLDR2, BESTDR5, MGC, SIXDF, NVSS, SDSS-EDR, UKIDSSDR1PLUS, PSSA, GLIMPSE and UKIDSSDR2PLUS were all backed up to tape. THE WSA ingest database was also backed up before the work on Ahmose and the system backups are going as normal. CU7 for the GPS is now being run for 05A." Hardware: Several issues with the catalogue load server arose earlier this week, and servicing of that venerable unit is taking place as this is being typed. JB notes: "Ahmose had a hardware failure in one of the RAID backplanes, this was replaced today but followed by a broken RAID controller, this was also replaced (along with a disk that had also failed previously) and Ahmose now seems to be back on it's feet." MSH reported that the opportunity is being taken to move things around in the racks while it is offline; the team also discussed a few suggestions for rearrangements and reconfigurations that may help to robustify the situation. Software: MAR reported working on the web interface and docs for the release. RSC reported: "Have finally got around to creating the new Schema module - an enhancement of the old Schema package to improve the schema driven features of the curation scripts. This solves a bug in SyncTestDb (see trac ticket 23), and makes CU16 even more schema driven. Next I'll be incorporating index, constraint and view creation features to the module in a consistent way, which will give CU19 the capability of creating views (see track ticket 25)." ETWS reported: "Refactored the parser to enable faster webpage upload in the browser and tightened the non-survey tables which resulted in cutting the size of the final tarball in half." NCH and RSC have started work on refactoring CU19 to make it much less disk-hungry and scalable for the GPS, and also for VISTA. Survey Data Release: NCH congratulated everyone on the release (albeit minus the GPS merged sources) of UKIDSS DR2. Much work has been done on the web site in addition to the business of getting the release out. Inevitably, thoughts are already turning towards DR3. There are several issues that are likely to vex all concerned: i) The data flow snafu from JAC - it is presently unclear if the 06B data from late Oct to mid Dec will ever be recovered, let alone how long it would take to sort out even assuming it can be; the hope is that after mid-Dec the problem is sorted (but we note that the data are arriving piecewise at CASU...) ii) PSF derived attributes are likely to become available at some point over the next processing cycle, and there will be undoubted pressure to go right back to the start and rererereretransfer/rerererereingest all catalogues from 05A to 06A (possibly even some 06B if we start transfering non-PSF catalogues soon?) No doubt from the user point of view, winter follow-up observing '07/'08 would be nice for 06B data... WFAU's take on this is that reingesting all of 05A to 06A is a huge undertaking in itself, and we would prefer not to have to do it, lest it badly delay the next DR3. Who wants PSF fitted attributes anyway? Oh yes, the GPS ... hmmmm. ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. Non-survey Data Release: Nothing to report this week. Astrogrid deployment: Now that UKIDSSDR2PLUS is out, MSH is asking around amongst AG about whether to substitute this in for the DSA DR2 resource. One issue is that only UKIDSSDR1 (and not DR1+) has been available up to now through DSA, and substituting with DR2+ may confuse users; apparently each instance of the secure DSA can only talk to one DB...? Miscellaneous: JB notes that he has now trained as a First-Aider. This means he will be able to resuscitate NCH next time the latter has a heart-attack over a data release....