From nch@roe.ac.uk Mon Nov 17 14:45:11 2008 Date: Fri, 14 Nov 2008 16:55:49 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archives weekly project meeting minutes, 14/11/08 Minutes of WFAU VDFS Science Archive meeting: 14th November 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, LGR, MAR, ETWS, RPB, NJC, RSC, MSH, PMW Apologies: JPE, JMS, AL, BCW, RGM DoNM: 10am, Friday 21st November 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH to contact MJI to suggest continuation of routine ops for 08B despite the known funnies in the deblends. Discharged; MJI (and AA) agree that we should keep up the momentum and not wait for the bug fix. ACTION: NCH to remember to ask ETWS to update the GLIMPSE tables for GPS cross-matching in DR5. Discharged; NCH checked and corrected the initiating script and he and RPB have run it on the ingest DB; any problems, blame NCH! Actions partly discharged but continuing: ----------------------------------------- None this week Actions carried forward from 07/11/08 meeting: ---------------------------------------------- ACTION: NCH to break everything by deprecating the loadServer DbConstant so that curations codes can be generalised to work for both VSA and WSA... but not until the WSA non-survey SW has been rolled out. - continues ACTION: All to comment on the development of the VSA metadata schema as described on the relevant TWiki page - continues ACTION: ETWS to add a delta-mag attribute to the VSA template detection table schema. - continues Specific points and new actions: -------------------------------- Project management: NCH noted that a(nother) half-yearly report has been prepared on WSA activities for the up-coming UKIRT Board meeting. PMW asked the team to fill in the progress reporting charts for October. WFCAM & VISTA updates: Nothing to report this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: Nothing new this week. WSA Operations: RPB noted a full backup of the WSA successfully done last weekend. ETWS noted: - CU4 has finished processing all 08A data (115 days), ingests are still going (34 days to do). The distribution of detections between surveys so far is: DXS: 1%, GCS: 5%, GPS: 39%, LAS: 13%, CAL&NonSurvey: 42% - I've updated the illumination table entries in Multiframe for WSA and UKIDSSDR4PLUS. - Freed some more disk space on the development disk and updated all DBs accordingly. RSC reported: - Updated CuPerformance on TWiki following final CU19 run. Total DR4 computation time after data ingest and QC was 36.5 days + another month or so for (one-off) recalibration. Hardware RPB reported: "Tests run on ahmose to ensure the disks are now behaving themselves. Everything checks out fine. Discussed with MSH prospects for upgrading one of amenhotep's jbods. Whilst it's possible with a few hours downtime of the live server, it was agreed in the meeting that it will be easier to upgrade one of the jbods on ahmose, despite losing 6 month's worth of maintenance contract. MSH suggested performing the upgrade when we move the servers into C2." The team agreed that upgrading the load server JBODs first (where we need the space) should be done regardless of invalidation of any residual maintenance contracts. MSH noted that moves to the new server room will be starting soon. The team discussed these in the light of the DR5 release schedule, and it's suggested that we delay WFAU server moves until after Xmas. ACTION: MSH to check with ITSG that WFAU server moves can take place early in the new year. RSC reported back from ADASS on cluster developments: "I've put my notes from ADASS 2008 onto the TWiki: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ReportOnADASS2008 The most interesting thing I learnt this year was about the GrayWulf cluster at JHU that links together 50 Microsoft SQL Servers to host PanSTARRS, SDSS etc. on a file system that totals to ~1 Petabyte. They are using the latest public version of MS SQL Server 2008, which clearly must have solved the performance issues with linked SQL servers and also be very well suited for dealing with cross-queries between databases. In fact, the databases must be split across servers. You can read the press release here: http://computerworld.co.nz/news.nsf/tech/0557AEED0B851CB7CC2574AC0076A467 I've also included in my TWiki notes, a response I received from Maria regarding the SQL Server design of GrayWulf. Essentially they've solved the problem of how to have a "parallel" SQL Server database - something that we'd really benefit from - but it really needs SQL Server 2008." MSH noted that he plans to visit JHU in Q1'09 to soak up some of their experience in deploying a clustered SQL Server set-up with MSSQL2008. Software: MAR has rolled-out the new colour image generation pages for internal testing: the new facility allows you to generate colour images from the shallow surveys. As you can generate up to a 9000 x 9000 pixel image involving the uncompressing of umpteen extensions the job emails the results. User-defined parameters include: low values (5-20) of stretch bring out background features, high values gives a more linear look. Clip=yes clips at the maxLevel rather than preserving colours by scaling by the max value in RGB. RSC reported: - Finished testing the bcp stdout-redirect workaround for gpsDetection outgests and committed software changes. - Revised deprecation methods in the automated curation code (runPreCU6) to ensure deprecated flags are consistent and removed the now obsolete deep image deprecation method in CU3. - Finished testing a complete deep non-survey release and committed software changes. - Reverted CU6 description in WSA.CurationTask to the original version, to clear up confusion between the role of runPreCU6 and CU6. - WSA.ProgrammeTable entries inserted for non-surveys that entered the WSA using the old CU21 method. - Further robustified metadata mirror update against the scenario where the catalogue server share paths are not working, whilst the databases are working - Currently merging three-month-old commits from release_7 into trunk and testing, so that a release_8 branch can be created, giving operations a cohesive set of software. Survey Data Release: NCH and NJC noted several discussions with relevant people concerning the DR5 release schedule. The team is very keen to get DR5, at least for the LAS, GCS and DXS, out of the door in time for the December UKIDSS meeting so that 09A follow-up of targets selected from data up to and including 08A is possible. There are significant worries that trying to do the deblend bug fix within this timescale as well is too much, so the suggestion is that we push back the fix to DR6. NCH is communicating with all interested parties to ensure agreement. Otherwise, DR5 QC will start early next week as 08A catalogue ingests should be finished this weekend. NJC noted that Seb Foucaud is up in Edinburgh next week, and we will have a chat with him concerning the next UDS release. Non-survey Data Release: RSC has been merging all recent software changes into SVN trunk, and once the 08A ingests are complete, the team agreed that retrospective ingests of all missing non-survey catalogue data should be done in order for a final end-to-end test of the new non-survey support regime for a few specific programmes (the suggestion is to do stellar ones first, and hold back on non-stellar programmes until the extended source deblended params are fixed). Astrogrid deployment & Data Analysis services No news on the Astrogrid front. On the Data Analysis front, LGR noted: "CASU's code is now on horus (ETWS, RPB). Correct rcore and nbsize values are used according to nustep (NJC). Illumination correction completed. Building infrastructure and investigating discrepancy of the order of 10^-2 mag (with respect to catalogued aperMags)." Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================