From nch@roe.ac.uk Fri Feb 22 19:01:17 2008 Date: Fri, 22 Feb 2008 17:02:30 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Nigel Hambly , Nicholas Cross , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 22/02/08 Minutes of WFAU VDFS Science Archive meeting: 22nd February 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, NJC, PMW, RSC, AL, LGR, BCW Apologies: JPE, MAR, JMS, RGM, MSH DONM: 10am, NB *Fri 7th March* 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH (following advice from MSH) to ask ITSG to move LTO-3 drive to the WSA network. Discharged; ITSG helpdesk ticket submitted; drive has been detached from the old server. ACTION: NCH to schedule a full WSA backup for the coming weekend. Discharged Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration after DR3. ACTION: MAR to take forward the possibility of archive and/or on-the-fly colour image production. Continues; line put in the forward plan for Q1'08; MAR has been studying renaissance masterpieces in the Florence Uffizi for inspiration... Actions carried forward from 15/02/08 meeting: ---------------------------------------------- ACTION: MSH to liaise with IT Support over set up of NFS file share from hatshepsut Continues; NCH noted that although lower priority, would be good to get this sorted so as to shift SegueDR6 onto hatshepsut (plus restore DB then SQL backup) so that the transfer files can be scrubbed from disk. Specific points and new actions: -------------------------------- Project management: Interviews for the operations post are to take place on 29th Feb with PMW, NCH, RGM and MSH on the panel with ETWS holding informal chats and showing the candidates around. NCH noted that JAC have reviewed their top-level archive requirements as part of the on-going MoU developments etc. and have asked for input from WFAU. AL noted that an internal meeting was held earlier in the week with PMW, NCH and MAR to discuss our position with respect to establishing a working interface with ESO-SAF for Vista data. AL is visiting ESO in early March for a VO review, after which a conflab will be held with Paolo Padovani and NCH, MAR by telecon to discuss the way forward. In the meantime, a local TWiki page will be established to assemble all relevant materials (formal SMP guidlines and conditions on delivery of data products, example ICD etc.) ACTION: AL to set up a WFAU TWiki topic on ESO-SAF interfacing (both technical and political...) WFCAM & VISTA updates: No news this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: NCH noted that users had noticed our FIRST catalogue was not quite what it should be earlier in the week; ETWS, RSC and NCH fixed the problem as quickly as possible. ACTION: NCH to bug MAR about release history/wsa-announce messages concerning the FIRST catalogue and associated crossmatch RSC noted that there is a TWiki page where we can put some notes on how to prepare external catalogues: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ExternalCatalogues WSA Operations: ETWS noted some worrying signs that there is an incompatibility with Debian 4.0 somewhere in the third-party software that is causing Python to seg-fault under heavy querying applications. Some suspicions as to Python middleware mxODBC which has not been updated. ACTION: NCH to look into obtaining/licensing the most recent mxODBC Python DB middleware. As a result of the above, curation operations subject to occassional reruns on updated servers, or running reliably on servers running older Debian. All upgrades are on hold until the source of the problem can be found. Some issues discovered earlier in the week over non-clustered indexes on *Detection tables have been identified and corrected by ETWS, RSC and NCH. Hardware New web server (horus) was delivered by Eclipse earlier this week, and has been set up by MSH. Software installations need to be specified for this. RSC noted that NFS mounts for the public servers are not available from khafre ACTION: ETWS to get ITSG to mount public server shares on khafre: /mnt/amenhotep and /mnt/thutmose ETWS noted that server khufu appears to have some wierd BIOS issue that is preventing upgrade to Debian 4.0 (blessing in disguise given the above). As a result, data hosted by this machine is to be shunted elsewhere in case it collapses in a heap at some inconvenient point. Copies are running at a respectable (given recent performance issues) 20MB/s. Software: RSC noted: * Some improvements to CU21 non-survey schema parsing strategy (documented in ModifyingSchemas on the TWiki) to update non-survey schemas for new indices. This will also aid us in updating the non-survey schemas during the recalibration development work. * Removed the static dependence on the load server share path from the Ingester/Outgester classes used by CU7, CU16, CU19 etc. this will allow us to test changes to these scripts for the schema redevelopment on test databases not on the load server. * The dev_1 branch was finally and successfully merged into trunk on Monday. By the end of today I will have created the new dev_schema_1 branch to work on schema changes for recalibration. * Minimised the number of calls to check constraints to avoid tempdb conflicts between development tests and operational runs due to a possible MSSQL Server bug. Tables without foreign key constraints no longer check constraints as it is unnecessary in this case. MAR noted that he has worked on getting the SSA interface running against SQL server 2005; a few problems with crossID and the collation but everything appears to be working apart from the thumbnail images which isn't a DB issue. Survey Data Release: RSC noted: * Created objID indices for *Detection and applied to released UKIDSSDR3 databases * Created new FIRST x LAS neighbour tables on WSA and all release databases on amenhotep, following investigations into why the original versions were wrong Non-survey Data Release: MAR reported that all registered nonSurvey users (apart from 08A onwards) now have flat file access). Astrogrid deployment: MAR, RGM and MSH have been working with ESAC folks this week. MAR reported that in collaboration with Aurelien Stebe and Pedro Osuna using the DAL toolkit, a SIAP service is now up and running for UKIDSS DR1; they also worked on a SSA (Simple Spectra Access) service for 6dFGS spectra. AL noted some high-level developments in VO-land. As part of IVOA initiatives, the ESAC folks mentioned above have been working on a generalised catalogue data model to act as a translation layer between users and rather complex data model schemas that exist in DBs such as the WSA. On a related subject, ETWS noted that the metadocs for all the DSAs are being reviewed as part of the on-going AG roll-out of new infrastructure software. Miscellaneous: Nothing else this week