From nch@roe.ac.uk Fri Feb 1 16:59:39 2008 Date: Fri, 1 Feb 2008 14:47:19 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Nigel Hambly , Nicholas Cross , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 1/2/08 Minutes of WFAU VDFS Science Archive meeting: 25th January 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, LGR, NJC, PMW, RGM Apologies: BCW, JPE, AL, JMS, MSH, MAR, RSC DONM: 10am, Friday 8th February 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: MAR to look after Windows system/DB backups prior to the new operator arrving Discharged ACTION: ETWS to examine CU21 to see if it can be tweaked to propagate template schema changes into pre-existing non-survey static SQL scripts Discharged ACTION: MSH to check RAID stripe sizes etc. on the Windows side as a check on file transfer performance when writing to 2TB LVs. Discharged ACTION: ETWS to JPEG and CU3 early 07B data asap Discharged ACTION: RSC to set up WFCAMOPENTIME with semester 05A as a test to enable flat-file access to non-survey data outwith it's proprietary date. Discharged; see Non survey below ACTION: MSH to set up new JBOD for batch server thutmose to receive further SDSS databases. Discharged Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration after DR3. ACTION: NJC to look after LAS/GCS recal issues following DR3 release. Continues; NJC has prepared file logs including non-survey following advice from RSC; CU8 needs to be run to apply the recalibration. ACTION: MAR to take forward the possibility of archive and/or on-the-fly colour image production. Continues; line put in the forward plan for Q1'08 ACTION: NCH to email the VISTA EPS PIs to get further input on database design for their respective surveys. Continues; NCH has circulated a draft email to the team; NJC and PMW have provided comments; will send out early next week after RSC has had a chance to have a look. NCH has spoken with VMC and emailed VVV PIs separately on related issues. Actions carried forward from 25/01/08 meeting: ---------------------------------------------- ACTION: MAR to incorporate basic info on using flat files Continues ACTION: RSC to coordinate migration of the residual CVS-tracked projects into SVN. Continues ACTION: MSH to liaise with IT Support over set up of NFS file share from hatshepsut Continues Specific points and new actions: -------------------------------- Project management: Recruitment of the operator position has been extended by one week; an interesting field of candidates has applied already. STFC have acknowledged WFAU's request for a quick note on likely support for VDFS ops in order to be able to offer the position for more than two years. The team completed the January progress charts during the meeting. WFCAM & VISTA updates: No news this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: RGM noted that Herschel Key Programme PI Steve Eales has contacted WFAU concerning support for catalogue products along the lines of existing "external" VDFS matched data. WSA Operations: ETWS reported: - CUs 1 to 3 has been run complete for August 2007. - Created tables for newly registered non-surveys and updated old tables according to the changes made to the schemas. - Included the number of total counts for each semester on the Monitor pages. - Finished 3rd party software installation on djoser. - Started work on the next phase of parallelisation concerning a faster ingest turnaround between CUs 1 and 3. NCH noted that we should keep up release momentum and notify SJW when the first month of data is ready for eyeball QC by survey heads. ACTION: ETWS to email SJW that the first month of JPEGs etc are ready for perusal by survey heads. MAR has done many tape backups over the past week. Hardware NCH noted that some RAID maintenance has been necessary on the catalogue servers this week; public server amenhotep has had all SCSI channels clocked back to Ultra160 and batch server thutmose will have the same set on Monday during the normal first-Monday-of-the-month W2K3 reboots. Eclipse delivered the new 48TB NAS box midweek, and this will need set up when MSH returns from holidays. PMW noted that he has been deep in conference with UoE procurement and Eclipse over getting their status upped to official supplier to smooth over any more tendering issues. RGM noted that IT Support have been asking IfA about their medium-term computing housing requirements, and the team agreed that WFAU's requirements should be treated separately. ACTION: PMW to check that IT Support have our future computing accomodation requirements factored into plans for the new room ETWS noted that djedefre threw a wobbly in the week, and that it was difficult to console log in as there are not enough kvm ports for all servers in the racks. ACTION: PMW to take advice from MSH on expanding the kvm units. Software: Regarding the latest Debian 4.0 installation on djoser, NJC is making detailed tests to ensure all is well with the software suite. NJC has been reworking schemas for the synoptic data and continue the curation software for this cu6. NJC has also been continuing to write a paper about archiving the synoptic data. Survey Data Release: DR3 phase-2 (i.e. including the GPS) finally emerged this week after some juggling of terabyte-sized DB files between the various SQL Servers by MAR. Web pages have been updated to reflect the latest release. Non-survey Data Release: Much communication this week with JAC concerning support for open time programmes. Chris Davis has been assembling a webpage doc to lead PIs through the procedure: http://www.jach.hawaii.edu/UKIRT/observing/cookbooks/wfcam_reduction_cookbook.html and MAR & RGM have been advising. NCH has enquired (prompted by MJI) about how we treat open access to UH and NOAOJ PI data - e.g. in the same way as PATT. Web docs and release DBs are in the process of fettling to support/clarify. Astrogrid deployment: LGR described progress on the generalised list-driven anaylsis procedure he is prototyping for implementation as a user-driven service. Miscellaneous: (Leaving this in from last week:) NCH noted that the NAM 2008 at Queen's University Belfast is now open for registration, with wide field surveys (incl. Gaia & PanSTARRs) scheduled for the Thurs/Fri Apr 3/4: http://nam2008.qub.ac.uk/ NCH and MSH will be going; NCH asked the other team members to consider attending and then we will co-ordinate who is to present what (deadline for registration for oral presentations is 29th Feb).