From nch@roe.ac.uk Mon Dec 1 00:36:30 2008 Date: Fri, 28 Nov 2008 16:16:02 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 28/11/08 Minutes of WFAU VDFS Science Archive meeting: 28th November 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, MAR, ETWS, RPB, NJC, RSC, PMW, MSH, LGR Apologies: JPE, JMS, AL, BCW, RGM DoNM: 10am, Friday 5th December 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH to rerun the non-survey assignment script for 08B. Discharged ACTION: RPB to badger ITSG to sort out the RAID maintenance SW on the Linux file servers. Discharged (see below for outcome) ACTION: RSC to create the new release_8 SVN branch Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: PMW to get a quote for a new NAS box a la takelot. Continues; one quote in but two more required because of various procurement rule changes. PMW noted that cost per TB is falling roughly in line with Moore's law (good thing too since this is what we budget for in the grant applications; on the other hand it's all a bit academic if we don't actually have a grant and have run out of equipment dosh...) Actions carried forward from 21/11/08 meeting: ---------------------------------------------- ACTION: NCH to break everything by deprecating the loadServer DbConstant so that curations codes can be generalised to work for both VSA and WSA... but not until the WSA non-survey SW has been rolled out. - continues ACTION: All to comment on the development of the VSA metadata schema as described on the relevant TWiki page - continues ACTION: ETWS to add a delta-mag attribute to the VSA template detection table schema. - continues Specific points and new actions: -------------------------------- Project management: NCH noted that we'd been gazzumped from our usual location for these meetings a second time. ACTION: PMW to check that we have a rolling weekly booking for the Vista Hut meeting room. WFCAM & VISTA updates: Nothing to report this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: NJC noted requirements from the GAMA project to use SDSS DR7, and NCH suggested we should review where we are as regards a possible upgrade from SDSS-DR5 to DR7. ACTION: RPB to check out available space per LV on SDSS catalogue host thutmose ACTION: NJC to check out the incremental changes in DR7 over DR5 ACTION: ETWS to check out availability of SDSS-DR7 split over several files/groups/volumes WSA Operations: ETWS noted: - 08A GPS data is now completely ingested. - CU4 is now running on all non-surveys of all semesters to ingest all the not yet ingested catalogue data. - Created JPEGs and ingested catalogue data for last day of 08A. - Started transfer of October 08B data from CASU. - Refined the page layout for the thumbnail pages according to last meetings discussion. RSC noted that we should do a monthly FlatFileFinalise update for newly ingested data. ACTION: RPB to run FlatFileFinalise as appropriate for recent ingests RPB reported: "MAR noticed some blank jpegs whilst QCing. I checked through carefully and found a whole bunch that needed sorting out. Deleted the blank jpegs and reran CU2 to recreate them. Most are now fixed, though some remain blank. Checking the fits file for these showed that there was simply no data for some of the detectors for those observations, thus I've concluded that all the jpegs that can be fixed are now fixed, and any that remain are not our fault. ETWS noticed a torn page on UKIDSSDR2PLUS on amenhotep. The copy on thutmose worked fine, so I replaced the .ndf file reporting the error with the copy from thutmose. This has fixed the problem. After badgering IT Support about the failed disk on djedefre, it turns out that they were right that there is no problem (which is very annoying!). In order to ensure there is some ongoing observation of the state of the RAID array, I've requested a nightly cron job to run the tw_cli command taking its arguments from a file owned by scos (allowing us to change what it does easily without involving IT Support). This has now been set up and will email results to RPB. ACTION: RPB: Make sure the cron job behaves as required, then request ITSG amend the cron job to email results to ETWS and MSH, as well as rolling it out across the rest of the linux servers. MAR noted that DR5 QC1 is now complete for DXS and GCS, with just the LAS eyeball deprecations to go (awaiting eyeball results from SJW). There are some issues regarding cross-semester QC for the GCS and checking correct deprecations and ProgrammeFrame allocations for the DXS - NCH suggested that he and MAR get together after this meeting. Hardware: PMW asked the team to reconsider the upgrade/maintenance of the catalogue servers following a special offer on a continuing maintenance contract from Eclipse Computing. MSH noted (following the meeting): "I've checked the prices, and they are basically as follows: RAID Card: £130-150 146 GB Disk: £170 I can't price the cost of the backplane until I can shutoff one of the JBODS, open it up, and get the make and model (they tend to be pretty stable though.) So a £450 maintenance contract is worth it if we end up needing 3 replacement disks over the year. On the other hand, we could consider performing a second upgrade of 146GB to 300GB disks on one of those JBODS if we need the extra space. They'd both go off maintenance, but then we'll have 16 more replacement disks for the two JBODS left with 146GB disks. And then both Ahmose and Amenhotep will each have an extra 1TB of disk space over what they have now." Software: RSC noted: "I've been working on the finishing touches for the non-survey release script: - CU19 now automatically creates a login user name for non-survey PIs to access their databases on release (same login applies to all versions of the database) and these databases are now evenly distributed across the volumes of the public catalogue server. So, on completion, the database becomes immediately accessible through the web interface. - Cross-talk flagging has now been implemented for non-surveys, on the basis that only frames with GCS/LAS numbers of bright stars in the field are flagged, otherwise it's too inefficient and ineffective. Plots of the bright star number densities for each of the surveys are available on the QualityBitFlags page on the TWiki. There's now also an additional mosaic provenance completeness check, which may also be overridden, prior to cross-talk flagging to make sure that all component frames are available and non-deprecated following reprocessing. - I created the new AutoCurate script, intended for curating deep surveys for DR5 onwards, which is based on the same code used by the automated non-survey release script and makes sure the various steps are performed in the correct order including the new variability curation procedure (CU6). NJC and I have been improving the ability to resume interrupted auto-curation runs and handle re-runs correctly, and we have documented this new curation procedure in CurationOverview on the TWiki." Survey Data Release: NJC noted new documentation developments for DR5 regarding the DXS/UDS synoptic features, and that we must remember to reparse the survey schemas prior to generating the parsed web site documentation. MAR noted that DR5 QC has progressed well and we should keep up the momentum and try to run some later DXS and GCS curation tasks over the weekend: CU7 will run in append mode for the GCS, and CU13 will run (under the control of RSC/NJC's new AutoCurate script) for the DXS. Non-survey Data Release: RSC noted that 07B non-survey data proprietary expiration occurs next week so an update to the relevant metadata DB will be necessary. Astrogrid deployment & Data Analysis services: MAR noted that he and MSH had a meeting together with the VO group earlier in the week to discuss forward support for VO integration through AG infrastructure. MSH has assembled a TWiki note with the details at http://apache.roe.ac.uk/twiki/bin/view/WFAU/PostAstroGridPlan Regarding a new tool to apply list-driven photometric measurements within different passband WFCAM data in the same field, LGR noted: "matched input vs output coordinates owing to the closest match, provided the distance is less than a few pixels (rounding within one pixel size is not sufficient). Debugging and investigating other float-like issues." Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================