From nch@roe.ac.uk Fri Feb 16 16:13:47 2007 Date: Fri, 16 Feb 2007 15:36:05 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 16/2/07 Minutes of WFAU VDFS Science Archive meeting: 16th February 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, ETWS, RSC, LGR, NJC, JB, MAR, MSH Apologies: JPE, JMS, JDT, AL, BCW DONM: 10am, Friday 23rd February 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: JB/MAR to install UKIDSSDR1PLUS on thutmose Discharged ACTION: NCH to check the GCS source merging for DR2 Discharged; everything seems to be OK. ACTION: NCH to forward VISTA EPS SMP drafts to NJC for additional scrutiny Discharged ACTION: RSC to point SJW to his test DB to check out the prototype implementation of cross-talk error bit flagging. Dicharged; in fact, thanks to a great deal of hard work on the part of RSC the crosstalk flagging has been included in the main DB with a view to inclusion in the current release. Testing is ongoing, but early indications are that it's working rather well. ACTION: ETWS to do some CVS administration to help NJC in socially responsible use of the CVS...! Discharged; CVS top-level projects now looking much neater. ACTION: MAR to review external catalogue cross-matches in prepartion for CU16 next week. Discharged Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues; all done save 2MASS, GLIMPSE and BestDR2. ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete. o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; NCH, RSC and MAR discussed this in the week and because of some subtle complexities it will be put on hold until after DR2. RSC has made some code changes to make the curation applications robust against this very minor schema change when it does happen. Actions carried forward from 9/02/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. Specific points and new actions: -------------------------------- Project management: NCH noted that all VISTA EPS SMPs are due in today at noon. Both he and NJC have been scrutinising the draft SMPs. NCH asked PMW to bring along the Q1 progress charts next week for the team to fill in. ACTION: PMW to bring along the Q1'07 progress charts next week WFCAM & VISTA updates: Nothing new to report this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: ETWS noted that Ani Thakar at JHU had asked for some network transfer stats given the various transfers of SDSS DRs over the last few years. ETWS reported: "Pure transfer rate for BestDR5 with sector was ~5.3MB/s and BestDR5_EFG with 12-threaded wget-ftp was ~2.7MB/s. So it was about 10 days for 2.3TB of BestDR5_EFG. The 4 lost files showed the same performance and were here in ~14h. Some older statistics: In 2004 the BestDR2 ftp transfer was done with ~2.6MB/s (but I don't know what tool was used). In March 2006 we transfered BestDR3 with the same script as used for BestDR5_EFG (multithreaded wget) but from an http address and had 6 parallel threads with 1.5MB/s each, which added up to 9MB/s for the whole transfer. David Hanley suspected at the time that this linear scaling was down to a bottleneck between Amsterdam and Edinburgh. So in retrospect it looks like an internet transfer outruns ftp and udp." JB reported: "Sam was in contact late this week and he had a chat with Dave Tinkler from UKERNA about the UKLight connection, apparently Dave was not happy about putting the link in place until early April. Sam is about to send an email to David Salmon (CC'd to David's boss, Dave and some people at our ends) pointing out the timescales involved as far as UKERNA is concerned." JB noted that he, PSB and Sam (Wilson; EUCS) have just about reached the end of their respective tethers over UKERNA and UKLight. Sam has indeed now sent a robustly worded email to UKERNA to try to get some kind of reaction. Them's fightin' words ... WSA Operations: JB reported: "System backups continued as normal. BestDR5 has now been restored and the flat files backed up. PSSA, SIXDF, NVSS, MGC, SDSS-EDR and UKIDSSDR1plus have all been copied to Thutmose and have, along with BestDR5, backed up to tape. BestDR5, MGC and SIXDF are also now attached to the SQL server. Provenance was done for the DXS after Eckhard ingested the deep stacks, Quality Bit Flagging was then done and CU7 run for the DXS, LAS and GCS. CU16 was then run for the DXS, GCS and LAS. Crosstalk flagging has now also been run for the LAS and is currently in progress for the GCS." Hardware: NCH noted that the JBOD for ahmose has been installed in thutmose's rack, but we might as well get another for ahmose anyway. This will all be sorted out after the dust has settled over the DR2 release. PMW noted that Eclipse are enquiring with Overland concerning installation of a second drive (LTO3) in our tape library. Software: NCH noted that CU19 has been checked for DR2 and all appears to be in place for creation of the release DB. MAR reported: "Added SDSS DR5 to schemas and neighbour tables and checked other external catalogue entries. Worked on script for updating web pages based on directories and images being added for Anaylis/Data Mining pages and plots. Finished updating UI forms to cope with views of DR2plus release." RSC reported: "Completed testing the adaptations to the cross-talk artefact flagging algorithm for deep stacks, and have overseen its application to DR2 released detection and source data. The script has been optimised, x,y indices were determined to be ineffective, and the latest information on reliability, performance and implementation has all been documented on the TWiki in the QualityBitFlags article. Tests on the GPS have revealed that the fields are so crowded that most sources would be flagged as cross-talk and the algorithm would take almost 100 days to complete. Therefore, application of the cross-talk flag to the GPS is impractical with the current algorithm parameters. Also, tabulated the performance of all the "run-up to release" CUs per programme comparing DR1 and DR2 figures - see the Twiki article CuPerformance." Survey Data Release: NCH noted that DR2 is now ready for the final DB preparation. As noted above, thanks to RSC crosstalk error bit flagging will now be included for LAS, GCS and DXS. Depending on the progress today on final runs of crosstalk errorflagging, CU19 (final release DB preparation) will be run either over this weekend or early next week. NCH asked that we now review the website documentation to make sure all is in place to inform users of the changes over previous releases. ACTION: MAR to switch off auto updates to the current website, ACTION: NCH to review glossary entries for all new attributes and procedures ACTION: MAR to create a error quality bit information page, and release notes update, with links to/from the glossary ACTION: NJC to add a Cookbook section concerning sample selection trade-off between completeness and reliability in the context of error quality flag thresholds. ACTION: ETWS to finalise Browser modifications necessary for DR2 Non-survey Data Release: Nothing to report this week. Astrogrid deployment: MSH requested that SDSS-DR5 be put online asap to enable AG access through DSA since apparently the Portsmouth mirror has gone belly-up. JB and ETWS noted that this will be done today since DB file backups to tape, and metadoc preparation, have both been done. Miscellaneous: NCH noted two conference announcements: NAM (Preston, April 16-20) and ESLEA (Edinburgh, March 26-28), and encouraged various team members to think about attending. Finally, NCH suggested early doors tonight at 6pm. Burrrrrp. N