From nch@roe.ac.uk Sat Sep 2 14:51:35 2006 Date: Fri, 1 Sep 2006 13:49:50 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU weekly Science Archive meeting minutes: 1st September 2006 Minutes of WFAU VDFS Science Archive meeting: 1st September 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, JMS, RGM, RSC, MAR, NJC, JB Apologies: JPE, AL, ETWS, MSH DONM: 10am, Friday 8th September 2006 in the Vista Hut meeting room Actions discharged this week: ----------------------------- None this week. Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS to finalise the browser pages for the two survey-ish non-surveys. Continues but nearly done; Transit survey is sitting on the public server and is ready for release. The following from last week partly done but continue: p) Release calibration and transit survey data as non-survey databases - ACTION: MAR, NCH & JB Continues; NCH and NJC have put the UKIRT FS data & associated metadata into the WSA in readiness. l) Review global archive schema to remove extraneous objects - ACTION: ALL, as we go along. NCH: VDF-WFA-VSA-002 Science Requirements Analysis PMW: " " " 003 Management and Planning NCH: " " " 004 Interface Control JB: " " " 006 Hardware/OS/DBMS Design NJC: " " " 007 Database Design MAR: " " " 008 User Interface RSC: " " " 009 Software Architecture Design, incl. local data flow RGM: " " " 010 VO Enablement ACTION: ALL to draft their respective documents (see above) by the 1st September. Continue; NCH noted that 5 out of 8 drafts have appeared; the rest (8, 9 & 10) would be checked in by close of play today. NCH suggested that we hold off editing each others documents for the time being until he has had a chance to overview the full set. MAR, PMW and NJC all noted that Quality Control is an issue that needs documenting at various points. Actions carried forward from 25/08/06 meeting: ---------------------------------------------- The following from last week continue: b) Merge in the development CVS branch to the main - on hold until RSC fully tested development software - ACTION: RSC d) Debug CU7 for deprecated frame sets and deletions on rerun - ACTION: NCH i) Enhance seaming to use quality bit information - ACTION: RSC (on hold until after h) k) Ensure ingest code can cope with 06A missing data quirks and any associated new attributes - ACTION: ETWS m) Insert Millenium Galaxy Catalogue as external survey; rearrange external catalogues onto batch catalogue server - ACTION: NJC & NCH (not JB as previously) o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH - all except b) are on hold until after the documentation for the review has been prepared. RSC anticipates doing b) within the next week or so, and notes that existing funcionality in the Main branch should (!) be unaffected. Specific points and new actions: -------------------------------- Project management: NCH reported that it looks like a VDMT will be held sometime next week in the absence of PMW. PMW will produce the usual progress material today, and hand to NCH for presentation at the meeting. JPE has indicated that he expects the meeting to concentrate on preparations for the FDR in October. The questions of a VSA logo, and the exact meaning of the VSA acronym have arisen over the past week. A quick discussion soon converged on "VISTA Science Archive" as the correct expansion of "VSA" as opposed to "VIRCam Science Archive" (c.f WSA) since the former has been used in the past and has got a certain amount of momentum (NCH), and also VIRCam is apparently not as official a name as WFCAM anyway (JMS). MAR is developing an aesthetically pleasing logo for web pages etc. WFCAM update: Nothing new to report this week, other than presumably MSB preparations will soon start for the next survey block (Nov'06 to Mar'07). Comments and issues arising from CASU fortnightly minutes: No new minutes as of 25 Aug AM. Networking: NCH noted with reference to last weeks minutes that CASU had indeed sent a correction Perl script for 3 05A nights - the problem is that raw Perl attachments get Horsted (i.e. embargoed) at this end for security reasons. Apologies to CASU; MJI kindly gzipped and resent to circumvent the KGB. JB noted that the NeSC UKLighters are still not responding to enquiries about what's happening. WSA Operations: JB reported: "CU1 seems to have run into problem with two last days of 05Av3, I have yet to trace it but it could be permissions based of some such. CU3 seems to have run fine for April of 05Av3, CU4 is currently running though at least one week will have to be redone due to an unknown issue with a dxsDetection ingest - a fix might be possible once the problem has been traced. The Transit survey is attached on Amenhotep. Ahmose has a hiccup but has now been fixed. IT is also having one of its drives consistency checked but more to be on the safe side than for any other reason (it's three other drives will be done as time allows). calDetection still has problems from its corrupted data page, attempted fixes have so far failed however there is another possibility that will be tried ASAP. Backups went fine as normal on Monday/Tuesday." Hardware: NCH noted the emails flying around this week concerning possible freebies from IBM, namely P690 shared memory machines going for a song. Two potential applications spring to mind in WFAU: i) performance comparison between MS SQL and IBM DB2 for Tbyte-scale DBs, and ii) CPU-intensive data mining applications when the new DM scientist joins WFAU in November. NCH noted that i) would require large disk arrays, and it has been hinted that these machines would not come with those; also that the power consumption/dissipation requirements would likely mean they should note be sited at ROE anyway. PMW noted the site services committee meeting next week, at which he would make sure this was flagged as a possibility, and NCH asked RGM to keep in the loop on developments ACTION: RGM to email Ken Rice (IfA contact concerning IBM P690s) about potential WFAU interest in high-end freebies from IBM. Software: RSC noted that he is now ready to merge the CVS development branch into the main branch to bring all documentation and code up to date and internally consistent for the review. The team agreed that this could be done next week without disrupting normal operations. No other news this week owing to work on documentation. Survey Data Release: NJC and NCH have bolted in the UKIRT Faint Standards to the WSA, and set up DB-driven metadata to create neighbour tables etc. for the calibration survey. A minor hiccup has occured in that SQL Server has managed to get in a mess over the housekeeping information for the calDetection table (RSC discovered that the row count is inconsistent between SQL queries and what's actually in there). NCH noted that neither Microsoft or Google Groups show any bug reports, advice or remedial procedures for such an eventuality and that probably the easiest solution is to bulk outgest the data, drop the table, then reingest, provided no solution can be found in DB Consistency Check (DBCC) calls. Non-survey Data Release: Nothing new this week. Astrogrid deployment: RGM noted that VOTech developer Mark Taylor (him of topcat fame) has asked for local JDBC access to some big DBs in order to try some optimisation tests on column-oriented access modes for very large tables. The team agreed in principle, but JB noted that we need to be a little careful over security considerations. MAR suggested we give him access to the SDSS-DR3 on thutmose via the same readonly login that the AG DSA software uses. ACTION: RGM to put VOtech developer Mark Taylor in touch with JB over local access to big SQL DBs via JDBC. Finally, RGM noted that authenticated access to proprietary datasets is being worked on actively within AG, and we should expect a release of DSA infrastructure around the end of this year to integrate WSA data into the VO via DSA. Miscellaneous: Nothing else this week.