README_roi56



I.  OVERVIEW FOR STANDARD SPECTRAL ROI (SPECTRAL PIXELS 56-167)

This is a description of the new procedure to process dark and flat data for the 
Hinode Spectro-Polarimeter.  The procedure expects flat and dark data to be taken
during the spacecraft eclipse season, such that paired dark (during spacecraft
night) is coupled with a nearby flat field observation taken during spacecraft
day.

Paired observations are to be taken with the spectral ROI starting at 56,
with the standard 112 spectral point sampling.

Separate sets may be taken in standard mode and fast map mode, but normally
only the standard (normal map) mode observations are taken, and fast map flats
and darks are inferred from those observations.  

During flats, the CT mirror must be dithered.  A significant number of observations 
of this type should be taken during the eclipse season, then averaged using the procedures
given here.  Typically around 10 orbits of data are needed to get a good average
flat field, with each orbit pairing a dark (during spacecraft night) with a 
flat (in the immediately preceeding or following orbit) observed at disk center.

Examples of this type of observational sequence may be found for 2009 July 29
and July 31.

I.A.  BULK PROCESSING

ROUTINES USED:
	AVG_DARKDATA.PRO
	COMBINE_DARKS_ROI56.PRO
	COMBINE_FLATS_ROI56.PRO
	FLATGEN3_SBSP.PRO
	GENFLT3_SBSP.PRO
	PROCESS_DKGN.PRO
	SELECT_FLATS.PRO
	SIZEOF.PRO
	SLITSHFTNP_SBSP.PRO

MODIFICATIONS WILL BE IMPLEMENTED IN:
	RECOVERGNDK_SBSP.PRO
	


Processing of multiple data sets to get dark then flat field data for each
dark/flat combination has been packaged together into one IDL/SolarSoft
routine PROCESS_DKGN.PRO.  A macro illustrating its usage with data
from 2010 is as follows (see runprocess_dkgn_std2011):

;======================================
;  macro to process flat/dark 2011 eclipse season normal map data 
;  first define input parameters
basedir = '/Users/lites/bruce/hinode/flatdark_july2011/'
flatdirs = basedir + 'flats/' + ['20110720_062120/','20110721_131715/', $
	'20110723_125235/', '20110724_133033/', '20110726_112732/', '20110727_120302/', $
	'20110729_113938/', '20110729_131814/' ]
darkdirs = basedir + 'darks/' + ['20110720_121507/','20110721_125807/', $
	'20110723_122807/', '20110724_130507/', '20110726_110322/', '20110727_114022/', $
	'20110729_111547/', '20110729_125505/' ]
npair = n_elements(flatdirs)
xctrs = fltarr(npair) & xctrs(*) = 0.
yctrs = fltarr(npair) & yctrs(*) = 0.
;  these are all the standard wavelength offset
roistarts = intarr(npair) & roistarts(*) = 56
;  process the flat/dark pairs
process_dkgn,darkdirs,flatdirs,roistarts,xctrs,yctrs
;  to skip selection of good images in case that such selection has 
;  already been carried out, use the following
;process_dkgn,darkdirs,flatdirs,roistarts,xctrs,yctrs,/skipex
;======================================

Notes:  

1.  The routine expects the names of ALL directories containing data 
to start with the string "20".  Users of these routines should name the
respective directories containing flats and darks to be identified by 
time of the first file in the sequence, as shown in the example above.
The level0 data corresponding to each operation, be it dark or flat, 
needs to be grouped manually into the directories so named prior to
beginning processing.

2.  The processing expects the dark and flat field data sets to be paired, as
illustrated in the example.  This ensures that flat field data is processed
with dark data that is very close in time.  If for some reason the dark
data of one or more operations are missing (or equally, the flat field
data), then one could substitute data more removed in time in its place
(for example, repeating two dark directories to maintain the pairing
of darks/flats).

3.  The procedure works equally well for Normal Map or Fast Map data;
see runprocess_dkgn_fast2010 for the corresponding fast map macro.
HOWEVER, fast map data will require input of a valid IDL-save file
from normal map data containing the information on spectral skew.  The
routine prompts the user for such a file.  FURTHERMORE, inspection of
the calibration of data with both the flats/darks generated from fast
map flat field data versus that constructed from normal map flat 
field data demonstrates that the latter is just fine, and that there
is no need in the future to acquire flats/darks specifically for 
fast maps.

4.  When reviewing and verifying the dark and flat field data, each
file must be verified as 'y' (to keep in average) or 'n' (to reject).  If
you make a mistake, you can "back up" (in the sense of retreating in
the sequence to a previous file) the verification by typing 'b'.  You
will then be asked how many files to back up. The index of the file 
sequence and the number of files in the operation being viewed are 
shown in the image display window.

5.  For dark images in particular it is important to reject all images 
with visible radiation hits.  These are usually most visible in each 
of the Q,U,V spectra, and can be lignt or dark in any of Q,U,V.  You
should look carefully at each image, some of the radiation hits are
rather subtle.  For the flat field data radiation hits are rarely visible.  
For the flats, it is important to reject data having dropouts (usually areas 
of uniform zero), even if they are in Q,U,V.  The code automatically checks
for zero in Stokes I and halts the procedure.  Type ".con" to continue,
then reject that file from the average.

5.  The output from FLATGEN3_SBSP.PRO (that calls GENFLT3_SBSP.PRO)
is a sequence of IDL-save files (and the SSW counterparts, .geny files).
One may process both standard and non-standard spectral ROI data, as
exemplified by the following macro where the ROI starting pixels
are specified for the corresponding dark/flat pairs for fast map data:

For the 2011 sample macro above, the output files are as follows:

goodarks_56_20110720_121507.save
goodarks_56_20110721_125807.save
goodarks_56_20110723_122807.save
goodarks_56_20110724_130507.save
goodarks_56_20110726_110322.save
goodarks_56_20110727_114022.save
goodarks_56_20110729_111547.save
goodarks_56_20110729_125505.save

goodflats_56_20110720_062120.save
goodflats_56_20110721_131715.save
goodflats_56_20110723_125235.save
goodflats_56_20110724_133033.save
goodflats_56_20110726_112732.save
goodflats_56_20110727_120302.save
goodflats_56_20110729_113938.save
goodflats_56_20110729_131814.save

shftavg_20110720_062120_roi56.save
shftavg_20110721_131715_roi56.save
shftavg_20110723_125235_roi56.save
shftavg_20110724_133033_roi56.save
shftavg_20110726_112732_roi56.save
shftavg_20110727_120302_roi56.save
shftavg_20110729_113938_roi56.save
shftavg_20110729_131814_roi56.save

avgflatimg_20110720_062120_roi56.save
avgflatimg_20110721_131715_roi56.save
avgflatimg_20110723_125235_roi56.save
avgflatimg_20110724_133033_roi56.save
avgflatimg_20110726_112732_roi56.save
avgflatimg_20110727_120302_roi56.save
avgflatimg_20110729_113938_roi56.save
avgflatimg_20110729_131814_roi56.save

gaintable_20110720_062120_roi56.save
gaintable_20110721_131715_roi56.save
gaintable_20110723_125235_roi56.save
gaintable_20110724_133033_roi56.save
gaintable_20110726_112732_roi56.save
gaintable_20110727_120302_roi56.save
gaintable_20110729_113938_roi56.save
gaintable_20110729_131814_roi56.save

gaintable_20110720_062120_roi56.geny
gaintable_20110721_131715_roi56.geny
gaintable_20110723_125235_roi56.geny
gaintable_20110724_133033_roi56.geny
gaintable_20110726_112732_roi56.geny
gaintable_20110727_120302_roi56.geny
gaintable_20110729_113938_roi56.geny
gaintable_20110729_131814_roi56.geny

6.  Once the processing has been done by FLATGEN3_SBSP.PRO, the dark/flat 
data from all operations may be merged to obtain the standard dark/flat
data for SP_PREP.  The merging is done by routines COMBINE_DARKS_ROI56.PRO
and COMBINE_FLATS_ROI56.PRO for darks and flats, respectively.
Examples of the macros used to run these two routines for the full set
of standard ROI=56 data for the 2011 eclipse season, are as follows:

;======================================
;  Macro to linearly combine (average) the operation-averaged dark
;  images for ROI56 of 2011 July NORMAL MAP data
basedir = '/Users/lites/bruce/hinode/eclipse_gndk_2011/'

darkz = [ $
'goodarks_56_20110720_121507.save', $
'goodarks_56_20110721_125807.save', $
'goodarks_56_20110723_122807.save', $
'goodarks_56_20110724_130507.save', $
'goodarks_56_20110726_110322.save', $
'goodarks_56_20110727_114022.save', $
'goodarks_56_20110729_111547.save', $
'goodarks_56_20110729_125505.save' $
]
darkz = basedir + darkz

combine_darks_roi56,darkz
;======================================



;======================================
;  Macro to linearly combine (average) the operation-averaged flat
;  images for ROI56 of 2010 May NORMAL MAP data

;  base directory path for results of flatgen3_sbsp
basedir = '/Users/lites/bruce/hinode/eclipse_gndk_2011/'
;  specify the gaintables
roi56list = [ $
'gaintable_20110720_062120_roi56.save', $
'gaintable_20110721_131715_roi56.save', $
'gaintable_20110723_125235_roi56.save', $
'gaintable_20110724_133033_roi56.save', $
'gaintable_20110726_112732_roi56.save', $
'gaintable_20110727_120302_roi56.save', $
'gaintable_20110729_113938_roi56.save', $
'gaintable_20110729_131814_roi56.save' $
]
roi56list = basedir + roi56list

;  specify the pathnames to the corresponding flat data
flatz = [ $
'goodflats_56_20110720_062120.save', $
'goodflats_56_20110721_131715.save', $
'goodflats_56_20110723_125235.save', $
'goodflats_56_20110724_133033.save', $
'goodflats_56_20110726_112732.save', $
'goodflats_56_20110727_120302.save', $
'goodflats_56_20110729_113938.save', $
'goodflats_56_20110729_131814.save' $
]
flatz = basedir + flatz

;  specify the string for identifying date of flat data
savnam = '_july2011_roi56'

combine_flats_roi56,roi56list,flatz,savnam
;======================================

The output files from this processing are in the SSW save format and
thus are ready for implementation in the SP_PREP /calibration/ archive:

darks_2011_56.geny
gaintable_july2011_roi56.geny

7. The routine RECOVERGNDK_SBSP.PRO needs to be modified
annually after the eclipse season flat/dark data are processed in order
to activate processing with the new flat field data.

The appropriate lines to update in RECOVERGNDK_SBSP.PRO are:
=======================================
;  standard normal map gaintables
;  note: the first entry below combines data from 2006-2007
;       so it is assigned year 2006
gaintbl_std = [ 'gaintable_stdmap_20070826.geny', $
				'gaintable_20090731_140100_roi56.geny', $
				'gaintable_may2010_roi56.geny', $
				'gaintable_july2011_roi56.geny' $
]
;  indices for years where we have standard spectral ROI gaintables
gn_yearstd = [2006,2009,2010,2011]
=======================================
adding in this example the 2011 data file name to gaintbl_std 
and the year 2011 to gn_yearstd

And also updating the dark level data base in RECOVERGNDK_SBSP.PRO:
=======================================
;  standard ROI normal map dark tables
darktbl_std = [ $
    'darks_2010_56.geny', $
    'darks_2011_56.geny' $
]
;  indices for years where we have standard spectral ROI darktables
dk_yearstd = [2010,2011]
=======================================
adding in this example the year 2011 to dk_yearstd, and darks_2011_56.geny
to darktbl_std.

8.  update the SSW distribution with the new RECOVERGNDK_SBSP.PRO to the
....../ssw/hinode/sot/idl/sp/util/ directory.
Also add the new .geny files
(in this example, aintable_july2011_roi56.geny and darks_2011_56.geny)
to the ....../ssw/hinode/sot/calibration/sp/ distribution.



I.B.  UPDATE THE RESIDUAL CROSSTALK DATA BASE

ROUTINES USED:
	FITSURF.PRO
	GENYCOEFF_XTALK.PRO
	CALIB_SBSP.PRO.NOICROSS

MODIFICATIONS WILL BE IMPLEMENTED IN:
	SP_PREP.PRO

The steps needed to get a revised residcross data set are as follows:

1.  Alter CALIB_SBSP.PRO to bypass the residual crosstalk correction; see
example of the altered routine CALIB_SBSP.PRO.NOICROSS in this directory where
OPERATION 10 starting at line 661 has been bypassed.

2.  Create an output directory for the calibrated data without residual
crosstalk correction, i.e.:
mkdir,/Users/lites/bruce/hinode/residcross_2011/cal_20110723_125235_noresid/

3. Run the standard SP_PREP.PRO, except use the altered CALIB_SBSP.PRO
from step 1, as per macro example run_spprep_20110723_125235.  Change
directories appropriately in this macro for the data being processed.
A macro demonstrating how to run SP_PREP.PRO is as follows:
==========================
raw_path = '/Users/lites/bruce/hinode/flatdark_july2011/flats/20110723_125235/'
restore,raw_path + 'filelist.save'
sp_prep,derf,l1index,l1data, $
outdir= '/Users/lites/bruce/hinode/residcross_2011/cal_20110723_125235_noresid/'
==========================

4.  Run the program fitsurf.pro on this calibrated data set.
For example:
IDL>fitsurf,'/Users/lites/bruce/hinode/residcross_2011/cal_20110723_125235_noresid/'

4.  Run genycoef_xtalk.pro to generate the appropriate SSW .geny file for
the residual crosstalk correction.  This program must be run under SSW.
An example of running the program is as follows:
==========================
;  input result from fitsurf
input_xtalk = '/Users/lites/bruce/hinode/residcross_2011/coef_xtalk.save'
;   output file name
outfilname = 'residxtalk_20110723'
genycoef_xtalk,input_xtalk,outfilname
==========================

5.  Modify the logic in SP_PREP.PRO to process data with the appropriate
residcross file.  The appropriate lines to update in SP_PREP.PRO are:
=======================================
yresid = [2006.,2007.,2009.,2010.,2011.]
;  define the corresponding yearly residual calibration crosstalk files
;  that are resident in the Hinode SOT/SP SSW calibration data base
xtalkfiles = ['residxtalk_20061026.geny', 'residxtalk_20071015.geny', $
    'residxtalk_20090731.geny','residxtalk_20100519.geny', $
    'residxtalk_20110723.geny' $
		    ]
=======================================
adding in this example the year 2011. to yresid, and residxtalk_20110723.geny
to xtalkfiles.

6.  update the SSW distribution with the new SP_PREP.PRO, and the new .geny file
(in this example, residxtalk_20110723.geny) to the 
....../ssw/hinode/sot/calibration/sp/ distribution.
Also add the new SP_PREP.PRO routine to 
....../ssw/hinode/sot/idl/sp/util/ distribution.

SOME NOTES ON HOW THE RESIDCROSS DETERMINATION WORKS BY EXAMPLE FROM 2010
FLAT FIELD DATA:

1. First calibrate 

/Users/lites/bruce/hinode/flatdark_may2010/flats/20100519_0206

one of the 2010 flat field data, with an altered CALIB_SBSP.PRO that 
bypasses residual crosstalk calibration.  These calibrated data are in:

/Users/lites/bruce/hinode/newresidcross/cal_20100519_0206

2.  Next run FITSURF.PRO, alter the routine to correctly read level1
data with READL1_SBSP.PRO.  Stores the fitted coefficients in
coef_xtalk.save

	-- get sinusoidal variation of Stokes Q in scan direction
	-- obvious linear variation of Stokes U along slit, linear variation
		in scan direction
	-- no obvious crosstalk in Stokes V, maybe a slight variation
		along scan direction

3.  SP_PREP.PRO restores the .geny file which evidently was generated by
GENYCOEF_XTALK.PRO, located in the ~/bruce/solab/residcross directory of old.
It seems to just transmit the parameters from the output of FITSURF.PRO:
coef_xtalk.save.  The items saved are [scoef,ndeg,sltpos,nftot], where
ndeg is converted to degn on reading the .geny file in SP_PREP.PRO.  These
items are as follows:

	scoef = coefficients of fit [ndeg+1,ndeg+1,3,nftot]
				(third index --dimension 3-- refers to Q,U,V)
	ndeg = degree of fit in focal plane image, linear fit in this case
	sltpos = slit scan step positions, dimensioned nftot
	nftot = total number of slit scan positions, 99 of old

So apparently the values are taken literally as they come out of the fits
and not smoothed as a function of the slit scan position

4.  the standard distribution program RESIDCROSS_SBSP.PRO linearly 
interpolates the values between the slit scan positions, but does
not smooth this variation
	
5.  Empirically I have determined that the boxcar smoothing in the slit
scan direction with a width of about 200 scan steps gives a good 
smoothed representation.  If the scan is full width (-1000,1000),
then this means that smoothing of about 

nsmooth = 200 * nftot/(max(sltpos)-min(sltpos))

or about 24 steps in the most recent case.  The routine FITSURF.PRO
has been adapted to do this.

The calibration program should work just fine with these coefficients
