From nch@roe.ac.uk Fri Mar 27 15:52:58 2009 Date: Fri, 27 Mar 2009 15:06:08 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 27/03/09 Minutes of WFAU VDFS Science Archive meeting: 27th March 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, RSC, PMW, MSH, RPB, MAR, RGM Apologies: JPE, JMS, AL, NJC DoNM: 10am, Friday April 3rd 2009 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH to feedback comments/requests to JHU regarding ports of HTM2 to Java/C++. Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS to parse that latest keyword info into the VSA metadata schema files. Continues; ETWS noted that communications with CASU indicate that the VIRCam data FITS keys are yet to be finalised, but that the procedure for parsing a standard set into an SQL schema is set up. ACTION: NCH to assign U/08B/14 H2/K frames to the GPS in the WSA. Continues; this can now be done as the corrected 08B data are now all present and correct (see below). Actions carried forward from 13/03/09 meeting: ---------------------------------------------- None this week Specific points and new actions: -------------------------------- Project management: Apparently we have in our hands (nearly) a piece of paper which details the bridge of the old WFAU grant from 01/04/08 to 31/03/09 in lieu of the new RG proper. This is timely, since the bridging grant period ends next week, so we have all of 2 working days to make spends on that year's worth of money - imagine how we laughed. Anyway, in order to invoice against this extension (as opposed to any new real grant we *may* get in the distant future) RGM/PMW suggested we buy any necessary small items (books/licences etc) over the next day or two. Let the good times roll. ACTION: ALL to spend like there's no tomorrow WFCAM & VISTA updates: No news this week. Comments and issues arising from CASU fortnightly minutes: With reference to the minutes of the meeting of 11/03, the team discussed the following: the team noted the progress on 09A and UDS; for the new sky frame keyword, this seems fine and very sensible, so if a heads-up & details can be given when it starts to be written in to the files, WFAU will make sure it gets propagated into the necessary DB table (Multiframe). ETWS noted he responded in the affirmative to the query regarding 05A->8A fixes; the team request that CASU send details of the procedure plus scripts so the same can be run at this end. Finally, on the question of partitioning of catalogue parameter #55 to accomodate a new attribute, the team expressed concern that this could cause trouble. While there is no problem in changing the DB catalogue schema for new data for WSA and all of VSA, we'll end up with a mixture of old/new FITS catalogues unless everything is rederived. (Incidentally, how can the FITS binary table #55 float value be bitwise partitioned?!). Note that WFAU ppErrBits flags near-edge detections and the generous WFCAM overlap plus WSA seaming in overlap regions result in the vast majority of such affected images being flagged as secondaries to better exposed primaries, so for that specific issue at least there's no real problem for users. Networking: Nothing of note this week. WSA Operations: RPB noted: "DR5 continues apace. SDSS-DR7 has been installed on one node of the new cluster. Although this is not accessible from the rest of our private network, it made it possible to outgest the necessary data for producing neighbour tables. The data was outgested, and the RequiredNeighbours table updated. Also updated the necessary schema files and saved them into SVN. GCS is ready for release. DXS is nearing completion. Source merging, reseaming, and neighbour tables are all complete. It's currently processing the variability statistics. LAS has had a wee hiccup (hiccough?). On producing the lasSourceXDR2PhotoObj neighbour table, it fell over with a torn page error. This may well be related to a previous problem we encountered after a restore from backup tapes where the DR2 flag table was corrupt. If so, that should be easily fixable and neighbour tables will be recreated. If not, we'll have to look into other possibilities. NCH mentioned the possibility of not producing SDSS-DR2 neighbours if the problem proved to be very hard to fix." ETWS noted: - Finished transfer and flat file ingest of the 08B reprocessed data. MAR noted that he has put info on the WSA website noting that the 08A/B flat file data are now all fixed. MAR reported: "Did some sanity checks of LAS and GCS post source merging. Steve OK with how LAS colours look though would still like to see a more robust way of doing the analysis." Hardware: For SDSS DR7, it transpires we don't quite have enough free space on the batch SQL Server. NCH's brilliant idea to stick a SAS box on the relevant machine (thutmose) has fallen foul of the antiquated expansion bus on that unit (PCI-X as opposed to PCI-Express) so the decision has been made to get another straight Ultra320 SCSI JBOD and run it at 160MB/s to avoid the nasties we've become familiar with over the last few years. PMW is busy getting N quotes where N is a large positive integer (to satisfy the UoE purchasing Stasi), so an order should be placed quite soon. Unfortunately this means the UKIDSS DR5 release will not be crossqueryable with SDSS-DR7 for a week or three, despite having the relevant neighbour table in place. MSH noted that he's worked around the cluster Windows networking problem; also has installed a new KVM switch (just needs the right cables for interconnect); and that a new private LAN switch is required as we've run out of ports on the current unit. Software: RSC reported: "Mostly been improving curation scripts and fixing the odd bug having identified some minor issues during the latest DR5 curation process. Notably the AutoCurate script can now redo a given release number and two CU7 scripts can now be run concurrently, which wasn't possible before, and has been used to good effect for the re-curation of DR5. Many of the new features will appear in the release_10 branch, which should be ready to create sometime soon. Have also been working with RPB on the necessary modifications to the software to allow database users to have read-only permissions but still be able to perform outgests." MAR noted work on extending login servlet for VISTA and adding email login details functionality. ETWS noted - Worked on the VISTA FITS keyword extraction code, but keywords are not finalised yet at the CASU end. - Created browser pages and metadata for VESPA database (see below) Survey Data As noted above, DR5 is *nearly* ready for release, consisting of LAS, GCS and DXS. NCH and MAR will do some final sanity checks over today/Monday. Non-survey Data Release: RSC noted that the WFCAMOPENTIME database was updated for new proprietary period extensions following a request from JAC. Astrogrid deployment & Data Analysis services: Nothing new this week. Miscellaneous: MAR noted "Submitted short WSA update article with pretty pics generated from colour image tool to UKIRT newsletter. Created and ingested VESPA database on thutmose for Rita Tojeiro / Ben Panter." Regarding the latter, RGM noted we should tart up the general web pages for non-WSA WFAU services to publicise them more effectively. MAR noted that browser pages have been generated for VESPA (son-of-MOPED?) so it shouldn't be too difficult to document that for more general use. Finally, MSH noted that he has now finalised the dates of his visit to JHU, where he plans to learn lots about SQL Server clustering from the SDSS/PanSTARRS team working there. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================