From nch@roe.ac.uk Fri Feb 10 16:22:22 2012 Date: Fri, 10 Feb 2012 15:19:24 +0000 From: Nigel Hambly To: Nicholas Cross , Eckhard Sutorius , Mike Read , Ross Collins , Rob Blake , Clive Davenhall , Mark Holliman Cc: Jim Emerson , Lorenzo Rimoldini , Stelios Voutsinas , Keith Noddle , Mike Irwin , Tom Shanks , Andy Lawrence at ROE , Norman Gray , Tom Kerr , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive team meeting minutes, 10/2/12 Minutes of WFAU VDFS Science Archive meeting: February 10th 2012 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: ETWS, RPB, NJC, ACD, MAR, MSH, STV, RSC, RGM, NCH, NG Apologies: JPE, AL, KTN, C37, 89X DoNM: 10am Friday February 24th 2012 in the VH (MAR in the chair) Actions discharged this time: ----------------------------- ACTION: RSC to provide details of the ESO VIDEO DR1 release compared with the proposed VSA release. Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: RPB/RSC to roll out latest curation client / DB server share configuration for routine ops use. Continues; RSC has carried out parellelised code tests and will make a code branch for ops testing by ETWS ACTION: RGM to liaise with Matt Jarvis over VIDEO public release. Discharged; uncertainty over whether new mosaics are to be produced and delivered to ESO for a new release with which the WFAU world public release should coincide, or if the existing ones that WFAU already has are the ones to use. NJC volunteered to try to find out. Actions carried forward from 28/01/12 meeting --------------------------------------------- ACTION: RPB to set up WSA mirror/sync for flat file DBs using MS SQL replication. Continues; the presence of mergelog tables in the VSA replication caused a clash with a survey release. So replication had been turned off! No-one could remember if mergelog tables needed to be in the sync DBs so they'll be removed and then see if that breaks anything. ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues. ACTION: RPB to complete the LTO-4 tape backups of VVV and SSA DBs and SSS flat file datasets. Continues; updated linux clients are being sourced to enable this to be completed. Specific points and new actions: -------------------------------- Project management: RGM noted that the grant renewal schedule is still somewhat uncertain, and that he needs to communicate with STFC over provision of hardware resources. WFCAM, VISTA and VST updates: No news to report this time. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 18th January 2012. There were a few comments: a) regarding the aperCor2 updates, RSC volunteered to add this to the growing list of things to retrospectively fix in the final archive releases; in the meantime, MAR has applied a one-off bodge to sort the problem in the recently reprocessed GPS DR8 data. b) it was noted that JRF will be joining CASU and that his algorithm for flagging areas affected by bright stars has raised some interest. Presumably this is the same algorithm that WFAU has been given, so if it is to be implemented in the CASU extraction/classification code somehow it would be good to know to save us duplicating the effort. Networking: All fine on the networking front. WSA/VSA/OSA Operations: ETWS reported: -- After running the difference image code for the GPS it turned out that we were missing the confidence IDs for the re-processed version4 data in the WSA. This is fixed now by a script that can also be applied when we ingest the data for the other surveys. -- Rerunning CU5 to produce the difference images now with associated confidence frames. This will finish tonight, after that we can proceed with the inclusion of the GPS into DR8. -- Released another non-survey successfully. -- ACD and I updated the schemas for the OSA from data from 4.5 months of observations. Recreated the updated tables and tried to ingest the data, but failed due to the large amount of different ESO programme identification codes used. We're awaiting Tom Shanks' input now before we can progress. Hardware and Systems: MSH noted updating the storage expansion TWiki page with the complete cluster capacity and a more accurate picture of what is available, noting also that we should probably make one of Ramses9, Ramses10, or Ramses12 a public server. Given the growth issues, and the fact we haven't really had a rethink of things in a while, MSH asked that interested folks convene in his office this afternoon to discuss the situation and plans. ETWS noted that he has finished the 3rd party software upgrade on curation clients shepseskaf and menkaure. Software: RSC reported: "Even more development and testing on CU19 related activities was required: A fixer script to sort out the DR8 metadata problem was created and released to ops; performance problems with the VISTA world release database production were resolved and the user interface improved to support field number ranges aiding ease of use for the VIKING release (this is now documented on the TWiki at CurationOverview#World_release). Source and synoptic source table outgests are now split into chunks for large VVV-scale tables to aid reliability. Finally the source of the function permission problem was tracked down and put to rest." RSC noted that the schema browsers seem to be out of sync with the SQL user-defined function definitions actually present in release DBs; this is a known issue concerning the static definition scripts. A clearer separation between functions common to all DBs and those that are survey-specific needs to be implemented in both scripts and the schema browser parser ACTION: ETWS to sort out common SQL function definition script(s) and ensure schema browser parser works correctly with those. Survey Data Release: VISTA public release update: VVV: still processing; ETA late March / early April (the team noted that it would be great if this could be done in time for NAM...) VMC: RGM suggested enquiring with the PI if a pre-NAM release would be acceptable to the survey team VIDEO: as noted above, a little uncertainty over the exact contents VIKING: all much clearer now as to the field contents, and this one should present no problems VHS: done and being checked (?) by the survey team; RGM volunteered to send a gentle reminder to the PI. NCH noted that some quick check CMDs and two-colour diagrams seem to indicate the data are of good science quality at least. NCH suggested that he and NJC get together later this afternoon to do some more thorough checks of the consistency of photometry in frameset overlap regions in both VHS and especially VVV, where NJC's relative photometric recalibration seems to indicate the presence of large offsets. Non-survey Data Release: NJC, ETWS and MAR have prepared one WFCAM non-survey release DB for a PI programme. NCH noted that non-survey private PI programme access for VISTA is likely to be required very soon ACTION: MAR to check that non-survey flat-file access for VISTA PI programmes is all in hand Astrogrid deployment & Data Analysis services: NG gave a brief description of the work that he's been doing concerning a facility to march prototype UCDs forward to the new "standard" UCD1+. The plan is to provide a live web service as well as something for internal use in migrating all static SQL schema script tags forward to the new definition. MAR noted that a few more queries had been raised with wsa-support concerning the list-driven photometry on-the-fly service offered through the WSA using CASU's toolkit code under-the-hood. All inconsistencies logged by users so far seems to be undocumented "features" of the code rather than bugs; thanks to MJI for responding rapidly to all concerns raised. Miscellaneous: ETWS and ACD have worked on the browser pages for Gaia-ESO Survey (GES) archive, and the prototype pages plus browser are currently being reviewed prior to going out to the consortium for input and comments. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.