From nch@roe.ac.uk Sat Aug 4 15:15:22 2007 Date: Fri, 27 Jul 2007 12:37:53 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 27/7/07 Minutes of WFAU VDFS Science Archive meeting: 27th July 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, PMW, ETWS, LGR, MAR, NJC, JMS Apologies: JDT, BCW, JPE, JB, RGM, AL, MSH DONM: 10am, Friday 3rd August 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: ALL to complete the EU questionnaire on e-Science digital archives. Discharged ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. Discharged; thanks to SJW for being entirely reasonable over this. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration. ACTION: RGM to look through requirements docs to see if any users have mentioned GALEX. Continues; RGM supplied GALEX details (included in last week's minutes) Actions carried forward from 20/07/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. - CONTINUES ACTION: AL to check on the exact wording of the survey release policy with reference to world release of proprietary survey data. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH noted VDMT is this afternoon at 3pm; NCH, PMW and JMS agreed to meet in PMW's office (PMW has circulated the usual reports). PMW noted that the VDFS extension grant is in the bank - development funds are now secured to end Q1'08. NCH noted that he and MJI have received a draft MoU from JAC Director Gary Davis; NCH has forwarded to AL, RGM and PMW for comments; PMW suggested JPE should see this also. NCH will collate all comments (thanks to RGM and PMW for responses; unfortunately AL seems to have disappeared on hols) and then reply to JAC asap. WFCAM & VISTA updates: NCH noted that WFCAM is back on telescope early next week; JMS noted that there is no change to the current VISTA schedule. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 27/07/07 am. Networking: Nothing new to report this week. WSA Operations: ETWS reported: "Transfer of the last missing days of 07A has finished. The compressed image creation for these days is well under way and the metadata has been extracted and is awaiting ingest. The extraction of data from the catalogues of the available 06B and 07A data has nearly finished. Ingest will take another 2 days. Parallelising the ingest CUs worked out very well. 3/4 of the 06B/07A catalogue data (~90 days) have been extracted in 3.5 days and at the same time ~60 of these days have been ingested. Under best conditions this would have taken 7 to 10 days with the old code." NCH thanked ETWS for these impressive developments, and the team expressed congratulations on the results. Hardware: Nothing of note this week. Software: NJC reported work on integrating the CASU-supplied list-driven photometry tool into CU9. Everthing is going smoothly so far; more testing needs to be done. MAR reported: "Worked on implementing new features to ImageList namely writing page results to a wget script file for nonSurvey use and making the number rows returned selectable up to 1000. Functionality in place just need to change access form. Had a look at the VO's footprint service to see if it could be useful in determining overlaps between surveys such as UKIDSS and SDSS." RSC gave an outline of his current work on organising code and branches in the SVN repository: "We've decided to start treating cirdr, the CASU source extractor, just like we treat other third party libraries. There is now a single, permanently compiled copy in the scos installation directory on each server that all users can access. It is now removed from the SVN repository because it was becoming difficult to manage there. I've completed testing of the software following the refactor of the CuSession class (see ticket 26 on trac), and all changes are now merged into trunk. Next I need to enhance DbSession to support forced connection dropping as well as persistent connections, which is required for the parallel CUs (1-4) - see ticket 68 in trac. Once ETWS has committed his latest bug fixes to CUs 1-4 to trunk, I shall create a new dev_1 branch to commit the DbSession changes to, which will then allow ETWS and me to update the parallel CUs IngCuSession class to make use of these features and to be derived from CuSession, like the other CUs. Following this I'd like to organise a software meeting on Monday or soon after, for the developers to discuss new software features that have entered the repository since the last meeting and to decide what will need testing (if anything) in trunk prior to creating a new release_2 branch. Whilst testing the CuSession changes I always took the opportunity to polish up CU5 and CU8, which were written rapidly for DR2 and haven't been looked at since. For CU5 I've created a testing strategy which is detailed in the TWiki SoftwareTesting article. Most notably CU8 can now update more than one detection table per multiframe if that multiframe contributes to more than one programme (with different detection tables) - see ticket 12 on trac. Also I've reduced the number of hard-wired constants in CU8 and the SourceMerger class by making them much more schema driven." ACTION: RSC, NJC & ETWS to meet on Monday 30th July at 11:30am for a SW architecture meeting. Finally, RSC noted that he has written some guidelines on working with temporary development branches and release branches based on his recent experience of SVN branches: http://apache.roe.ac.uk/twiki/bin/view/WFAU/UsingSVN#Branches NCH noted that he has robustified the seaming code against whacky data that creeps through QC; this should significantly speed up GPS seaming. RSC kindly volunteered to integrate these changes into the version in the SVN. Survey Data Release: The team recalled that the EDR is to go live on the 10th August: ACTION: MAR to switch off proprietary status of UKIDSS EDR in the WSA interface on August 10th. Much discussion this week centred on DR3. NCH reported back from a constructive conflab with SJW earlier in the week, where the Consortium Survey Scientist expressed his wish that calibration tweaks at the 1% level are not a priority and we should proceed with DR3 preparations as fast as possible. RSC noted some possible enhancements to quality bit flagging (UDS q-bits; frame-edge funnies and proximity to bright stars) can probably proceed in parallel with transfer/ingest of the remaining (early 06B) data. Responsibilities for DR3 preparations are as follows: MAR (& NCH): Survey QC NJC: liaise with the UDS folks ETWS/JB: early CUs for early 06B followed by later CUs RSC: Q-bits and miscellaneous SW support. ACTION: NJC to contact the Nottingham UDS folks to find out their intentions and timescale for DR3. The team discussed a tentative, highly conservative schedule for DR3 preparations. Based on the assumption (possibly incorrect, but let's be very conservative) that the early 06B data is released to us in a single chunk at T0, the following schedule (in weeks) will follow: T0+ What: 3.0 CU1 (transfer - assuming sustained 3MB/s) 1.5 CU2 (jpegs) 1.0 CU3/4 (ingest and provenance updating) 1.0 CU8 (photometric zeropoint recalibration) 2.0 QC1 (much work to be done in parallel with previous tho') 1.0 Quality bit flagging 3.0 CU7 (source merging from scratch again, unfortunately) 2.0 Final CUs. which adds up to a somewhat disheartening 3 months. However, the first 5.5 weeks can be reduced if the 06B data is released in small chunks as it's processed (but we understand if this is not possible). Also, the degree to which the above tasks run in parallel has probably been underestimated, and GPS curation should be significantly faster than before since the schema has been slimmed down - NCH asked NJC to contact the GPS Survey Head about the final details: ACTION: NJC to contact GPS survey head concerning apermag 2/6 attributes in the merged source schema. Watch this space for further updates... Non-survey Data Release: Nothing to report this week. Astrogrid deployment: MAR reported: "Looked at saveToMySpace function, indeed it is now properly broken (things have changed/moved in AstroGrid). Should be able to produce something more stable based on JDT's PLASTIC applet. Also might be possible to use similar code to improve sending files to TOPCAT." Miscellaneous: ETWS noted receiving an email invitation to the Edikt2 technical workshop on data management at the Western General on Sept 4th; the organisers have requested a 30 minute contribution from WFAU on our activities and experience. The team felt it should be up to ETWS/JB if they would like to take part. Finally, PMW welcomed back NJC following his recent nuptials, and suggested we fix a date for an evening meal to celebrate. ACTION: NJC to liaise with his other half over possible dates for a celebratory evening meal.