From njc@roe.ac.uk Fri Jun 18 13:45:31 2010 Date: Fri, 18 Jun 2010 12:08:06 +0100 (BST) From: Nicholas Cross To: Nigel Hambly Cc: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Rob Blake , Ross Collins , CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr , Nicholas Cross Subject: WFAU VDSF Science Archive weekly project meeting minutes, 17/06/10 Minutes of WFAU VDFS Science Archive meeting: June 17th 2010 Held a day early, outside in the sunshine ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: RSC, ETWS, RPB, NJC, MAR, MSH Apologies: JPE, KTN, NCH, AL, RGM DONM June 25th 10am, NCH back in charge, so no sunshine. Actions discharged this week: ----------------------------- ACTION: RPB to ask ITSG about the impact of site power-downs and if necessary inform the community via our webpages and announce lists of any interuptions to normal service. Actions partly discharged but continuing: ----------------------------------------- ACTION: ALL to review new hardware requirements/procurement over the next couple of weeks Proposal is for one new curation server and one new server dedicated to web application running (though not necessarily a web server). This will be discussed with KTN next week. MSH knows what is needed - a new seshat and a new shepseskaf, but hasn't discussed this with KTN yet. ACTION: RSC & NJC to document new operational procedures on the TWiki Rough notes done but needs review by NJC and polishing for ops use. NJC has checked some of this, but not all CASU have released some prototype tiles to WFAU for ingest tests. ACTION: ETWS to try some test ingests of VIRCam tiles. ETWS has started to look into this as a top priority ETWS and NJC found a bug in the totalExpTime code which affects stacks, but not tiles. While trying to anticipate tiles an error in the logic was made. This has now been fixed. Other tests still ongoing. ACTION: NCH to fill out a Q2 2010 plan and continue with the progress monitoring for the time being. Continues ACTION: RPB/KTN to order several crates of LTO-4 tapes. Continues ACTION: ETWS to enhance the early CUs to cope better with CASU's process versioning Continues Actions carried forward from 04/06/10 meeting ACTION: TBD to draft and MOU on behalf of PSPIs for ESO-SAF deliveries (Action to hire TBD first) Specific points and new actions: -------------------------------- Project management: No new items WFCAM & VISTA updates: New ESO press release on VISTA: NGC 253. Comments and issues arising from CASU minutes: No new minutes but ETWS reminded NJC about the distortion and illumination corrections for VISTA tiles. NJC has emailed Jim lewis and Simon Hodgkin about these and has had no reply so far. ACTION on NJC to continue to try to find out what photometric corrections will be applied to tiles. Networking: ETWS noted that networking has been pretty normal: some outages, but no more than usual. WSA/VSA Operations: RPB reported: "WSA 10A data now fully ingested up to the end of April (which is as much as we've currently got). VSA P85 data ingesting continues. Now that we can run curation and ingests at the same time, and run curation in parallel, the ingest will become the bottle-neck, so we've been looking at ways to speed this up as much as possible. Hopefully ordering the data by primary key will help speed things up a bit. " Hardware and Systems: MSH noted: Putting together new orders (see Actions partially discharged) and updrading shepseskaf to the new version of Debian. This should hopefully fix the SVN bug. There was a brief discussion about other third party software issues that may be fixed by this. NJC brought up the issue of SExtractor. ETWS assured him that there was a new installed copy on shepseskaf. ACTION on NJC to test SExtractor on shepseskaf and finish off CU13 upgrades Software: RSC noted: CU16 now has support for date ranges for synoptic survey curation. Updated release branch with changes to command-line arguments with compulsory date ranges and all of the recent bug fixes to synoptic survey curation etc. Documentation on the TWiki has been updated to reflect the command-line changes. Tiles are now supported by ProvenanceFiller and the non-survey deblended detection parameters fix is now complete following additional bug fixes to the fixer script. ETWS noted: -- Updated CU2, 3, and 4 to process tiles. Remember that all keywords are in the secondary in compressed tiles. -- Included the creation of a flag file to indicate that FinaliseFlatFileIngest has been run to ensure everything is in place before catalogue processing. -- Moved the code to update the illumination table entries into FinaliseFlatFileIngest to enable parallel CU4 processing and ingest by minimalising the database access for CU4 processing. -- Started to rewrite code to allow for non-integer versions. NJC has added in more indices for SynopticSource tables, and has fixed the logic in the pipeline which determines whether neighbour tables are recreated or not. NJC has tested Tamas Budavari's method of halfspaces for determining whether an object is in a frame or not and has starting implementing it, which should significantly reduce the time spent on the slowest part of CU6. RPB suggested using the MAXDOP 1 suggestion on bulk insert as this was still using parallel processing. ACTION NCH to decide on frame set tolerance for VVV source merging as their is a disparity between the documentation and the database. WSA/VSA user concerns: Request for epoch merged passbands from Stuart Folkes in Chile. This could be dealt with using a SynopticSource table, but this would be very inefficient when it comes to the single filter phase of the survey. NJC suggests having SynopticSource with a narrow date range and having two BestMatch tables (but only one variability derived from the SourceXDetectionBestMatch). This could be accomplished with few changes, but we are still not sure of all the implications. Maria-Rosa Cioni had many queries this week about the VMC. One is to do with crossID for the SynopticSource table. MAR said this was possible, but needed the correct indices defined (which NJC has done). She also asked about the pointings that observers use to define their surveys. NJC has clarified what this is with Jim Emerson and WFAU will try to make this simpler to find in future releases. She is also worried that the VSA is too complicated and her team do not know how to use it. NJC will phone into the VMC team meeting on afternoon of 28th June to go through some of the types of queries that the VMC team would like to do. We feel that a workshop for all VSA users would be useful. ACTION NJC to suggest workshop at next PSPI meeting. University of Hawaii users had queries about epoch merged bandpasses for 4 of their non-survey programmes in the same field, but different semesters. RSC, RPB and NJC have but a small amount of effort into creating a single programme which merges Survey Data Release: Non-survey Data Release: 5 being processed included the combined semester programme with correlated band-passes. Astrogrid deployment & Data Analysis services: Nothing to report this week. Miscellaneous: MSH suggested an early doors next Friday due to the beer festival and because Brian Walshe will be visiting. ALL want a good result in the football tonight. > ============================================================= > Nigel Hambly Tel: +44-131-668-8234 > Institute for Astronomy Fax: +44-131-668-8416 > School of Physics and Astronomy > University of Edinburgh Email: nch@roe.ac.uk > Royal Observatory > Blackford Hill > Edinburgh EH9 3HJ > > The University of Edinburgh is a charitable body, registered > in Scotland, with registration number SC005336. > ============================================================= > > > > >