From nch@roe.ac.uk Fri Oct 28 14:38:41 2005 Date: Fri, 28 Oct 2005 12:39:01 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes: 28th October 2005 Minutes of WFCAM Science Archive meeting: 28th October 2005 ------------------------------------------------------------- ------------------------------------------------------------- Present: NCH, RSC, ETWS, JB, MAR, NJC, AL, JDT, JMS Apologies: MCH, RGM, PMW, JPE DONM: 10am, Friday 4th November 2005, plate library Actions discharged: ------------------- None this week. Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 28/09/05 meeting: ---------------------------------------------- ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES; see note in CASU minutes below. ACTION: PMW to try new cross-ID tool - CONTINUES Specific points and new actions: -------------------------------- Project management: JMS reported that the next VDMT would be held on the 8th November at 2pm. Some uncertainty as to whether this is a telecon or face-to-face; JMS will not be attending, so he suggested NCH clear up the logistics with the management team. ACTION: NCH to contact VDMT concerning the Nov 8th meeting. AL and NCH reported attending a UKIDSS survey heads telecon on Wed 26th. Various issues concerning engagement with the SV community were discussed; NCH noted that as far as the DXS and UDS were concerned, ad-hoc access to flat file products had been been arranged previously in advance of any prepared database release, and this would continue to be the case. Otherwise, the main issue of relevance to the WSA was of course the release timescale - see Data Release below. PMW and NCH will be meeting on Mon 31st to assemble the Q4'05 plan (delayed by NCH on leave and PMW completely immersed in the GAIA proposal submission - see below) and also the Q1'06 plan. This needs to take into account, amongst other things, preparations for the V4 design phase and associated review. ACTION: NCH and PMW to meet on Mon 31st to assemble the Q4'05 and Q1'06 VDFS work plans Finally, NCH noted that a VEGA-GAIA consortium has submitted a proposal to PPARC last week which seeks funds (in part) to follow on from VEGA-VDFS and produce a dataflow system for GAIA photometric data, in the pre-launch period Q4'06 to 2011. WFCAM update: See the stop press pages linked from the TWiki for latest news. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meetings of 10th and 24th of October. The issue of the lack of a release timescale in the WFAU minutes is addressed below in these minutes; the team also noted the other main area of mutual interest, viz. that related to data transfer, where CASU have been investigating the cost and charging model - see Networking below. Finally, NCH noted with interest the alternative hardware solutions being investigated by CASU for mass storage (fibre-channel RAID brick with SCSI front-end). Networking: With reference to data transfer costs, UKLight is being investigated at both ends as a possible alternative, that will at the same time give increased bandwidth for scaling to VISTA data volumes. JB noted the meeting that took place between NeSC, UKERNA, WFAU and ROE IT Support networking folks last week. ACTION: JB to put some notes on the WFAU Networking TWiki topic from his own (and PMWs) meeting notes concerning UKLight. Transfer of SV1+2 (in fact everything up to and including May 20th) was completed (bar a very small number of problem files) on the 12th October. Finally, NCH noted that he had, by chance, discovered that our online SQL DB copy of USNO-B1.0 seems to have corrupted data in it (e.g. objects with Dec exactly equal to -90.0 degress; mags fainter than 25th, etc.). The degree of corruption is unknown at present, it being difficult to cross-check against a reliable source. NCH noted that we spotted a small degree of corruption last year, and traced it back to bad data supplied from LEDAS in Leicester; it could be that we didn't spot all the problems at that time. JB reported that he has been investigating, by restoring tape backups and checking for strange numbers in LEDAS's copy of the dataset via their BLASTA service. Anyway, it seems clear that this DB should be taken offline until further notice (as far as we are aware, nobody is using it anyway). JHU need to be notified of the problem, since they took a copy of our USNOB SQL DB. ACTION: JB to detach DB USNOB from public SQL Server on amenhotep ACTION: MAR to put notes on the SSA and WSA web pages to the effect that USNOB has found to be corrupted and will remain unavailable until further notice. ACTION: NCH to notify Jan Vandenberg at JHU that the USNOB SQL DB has been found to be corrupted. WSA Operations: JB reported: "Ahmose, Amenhotep, Djoser, Sneferu and Khufu backups to disk and tape for this week finished yesterday without problems. Worked through the CU3 problems ETWS (NANs, spurious FOCUS files and broken data from April 11th and 13th), all files excepting three of the four NANs have been removed as a solution to the problems, the final three NANs have been fixed by CASU, retransfered and have CU3 rerun on them." Hardware: Reconfiguration of the public SQL Server (adding in 2TB more disk) will take place on Tues 1st Nov (confirmed by Eclipse). In readiness for this, and because the server will likely remain offline for a few days, the existing UKIDSS release needs to be copied on to the load server and the interface tested for connectivity with it. ACTION: MAR & JB to attach a copy of the existing UKIDSS release to the load server ahmose and check interface connectivity in advance of the public server being taken offline next week. ETWS reported that software installations on khufu (under Debian Sarge and most recent g++ compilers) is progressing. JB reported that all is well with the various servers and RAID arrays, despite two consecutive power cuts on Tuesday (thank the gods for UPSs, once again). Software: NCH reported doing a small amount of work on CU19 in readiness for non-survey data release; also setting up a post-ingest quality control script to cope with generic and specific quality issues, as and when they crop up. RSC reported working further on CU16 and the integration of the fast plane-sweep algorithm; MAR reported working on the user interface in readiness for non-survey release and wrt various small tweaks as suggested by the UKIDSS SV community. NJC has mainly been on leave, but reported more work on the calibration data model (also relevant to general multi-epoch photometry), and has also been looking further at CU13/14 and integration of the CASU tools (e.g. catalogue extractor). ETWS has been doing much work on changes required by reprocessed data in CU1-4, and has been installing the curation software suite (including third-party code) on the newest storage node khufu. Survey Data Release: Transfer and ingest of SV1+2 data is now complete, and we hope to release these data to the UKIDSS SV community by the end of next week. With reference to the UKIDSS telecon attended by NCH & AL earlier this week, SJW hopes to declare SV complete 4 weeks after this impending release, and thereby give the go-ahead for release of the 05A May/June data to the entire ESO community in January (assuming there are no major issues needing substantial correction). Non-Survey Data Release: NCH noted that increasingly irate (and increasingly frequent) emails are being received by WFAU (and JAC apparently) concerning availability of non-survey datasets. We are trying to remain calm in the face of often rather unjust implied criticism; NCH has suggested to AA a release target of the end of November for Semester 05A (MAR and NCH are working as quickly as possible on this). In the mean time, we are making ad-hoc arrangements for PIs to access flat-file pipeline products once they have been transfered and ingested (i.e. prior to release of prepared, queryable database products). Astrogrid deployment: JDT reported that the WFAU AG portal is now up and running. He plans to now set up Community, Registry and the MySpace services (in that order). AL asked about the status of integration of the WSA into AG infrastructure, from both the WFAU and portal websites. JDT and MAR have developed a "save to MySpace" button for WSA interface-generated results sets, this just needs to be switched on. ACTION: MAR to switch on the "save to MySpace" WSA Interface button. >From the AG portal side, JDT suggested he could set up a "Datascope Launcher" for the WSA. ACTION: JDT to set up a Datascope Launcher for the WSA on the AG workbench pages. Miscellaneous: NJC noted that a calibration working group meeting is to take place soon (possible meeting on the 4th Nov has been rescheduled?). RSC noted that his report from ADASS is now on the TWiki at http://apache.roe.ac.uk/twiki/bin/view/WFAU/ADASS2005Report and that he will distribute his draft poster write-up for perusal before submitting it for inclusion in the conference proceedings.