Must-do checklist (that one might quite easily forget !) :

•  Turn on the bias voltage for the detectors.

•  Check the nominal insertion time (on the Target Pet page) for non-Emittance scans (like ~0.5 s) to make sure that it's not the insertion time for Emittance scans (which is much later ~1.2 s).  More below ...

•  Is the linear position for the target optimal (checked by observing the BLM readings when the target moves in and out, eg. in emittance scan) ?   If not, the beam (esp. beam sizes are typically big at the beginning of the run) may hit the target frame.  It may give multiple peaks in emittance scans and more noise covering a larger portion of the bananas at higher TOF.

•  Check that the RF Harmonic is set correctly for the appropriate "PPM user" ("pet"  → "AGS"  → "Polarized-protons"  → "RFparamsForPolarimeter").

  • To see what Harmonic AGS is in, check (from Keith Zeno): "pet"  → "AGS"  → "Rf" → "RF_Operations" → "HarmJumpControl" → Choose the correct PPM User → (depending on how it's set up) probably the "Harmonic #" on the "RST" column for most of the cavities (as some cavities may not be used).

•  Reminders:

  • The script/java program which shows the AGS polarization measurement results is called "krisch".

  • Only if you start the "PolarControl" from the user "mcr", you'd see a "CNI Analysis" button in the "Target Measurements" page, which you can use to view Andrei's analysis results such as "bananas" using the "Color plot" button.

What could go wrong during commissioning or even later ?

1. A hung "cnipol" process is preventing further polarimeter action
   
   For some unknown reason, once a while, a hung process in the polarimeter PC (cnipol3 ~ 2015) may refuse to stop/finish.  So, "ssh" to "cnipol3" and enter the following :
   
   ps -ef | grep cnipol
   
   You may see something like ("cnipol_noprog" in the example below may be just "cnipol") :
   
   mcr  27095 27091  0   08:25   ?   00:00:00 /home/mcr/ado/cnipol/cnipol_noprog -m -n65578 -d001 -u4 -e40000000
   
   If you found the above when nobody was taking any measurement, you should kill the process by doing:
   
   kill 27095
   
   where "27095" comes from the first no. in the output line of "ps -ef | grep cnipol".
   
   
2. How about the bias voltage for the silicon detector ?

   Check the bias voltage and current.  At the time of writing (early 2006), it's by looking at the TV monitor in HITL2 (Bldg. 908).  If it trips, it'll be zeroes all over the places and the CNI polarimeter system certainly wouldn't work if the bias voltage trips.  Here is a set of analysis plots that was made for a run when the bias voltage tripped.

   Starting from the 2007 run,  one can turn on the bias voltage from "pet"  → "AGS"  → "Polarized-protons"  → "CNI_HvBiasControl".   I've also observed that after the read-out rate got very high, the rate would decrease gradually over time if it was not used.

   • On the pet page, if the "OutPut V" and "OutPut I" are with some decent non-zero values but the currents for the 8 individual detectors are all zeroes, check whether the Scan button near the bottom of the pet page is On or not.
      
   • In case of problems, one may go to the bottom of this pet page and change for "PS" or "DVM" from "Remote" to "Offline and then back to "Remote".  If this doesn't work, one may go to "https://startup.pbn.bnl.gov/servers" → "inj_managers" "  → "polarBiasManAGS AGS Polarimeter Bias Manager" to restart the "Bias Manager".
      
      
   • If this still doesn't work, one may "ping gpib52.pbn.bnl.gov" to whether it responds.  [ There is also a utility called  "gpibtst" (in acnxxx machines) that one can put in "52" as the "GPIB #" to check statuses. ]  If not, there is 50-50 chance that we may need to AC-reset the network switch or the GPIB.  But try to AC-reset the network switch first.  Follow the following steps to AC-reset:

     Execute the command "fit" in (eg.) acnlinxx → click "Others" → click "Name" and look for AGS-CNI-POLAR-* stuff.  After putting the cursor on any of them, one may hit "AC reset" to "AC-cycle" (ie. to turn them off and then on).  There are 5 of them:
      

     • AGS-CNI-POLAR-HVSUPPLY is for the SRS power supply controlling the bias voltage of the silicon.  Be careful here !  Because the voltage may ramp up in a fashion which is too fast when the power is turned on.
        
     • AGS-CNI-POLAR-GPIB-ENET is for the GPIB unit which is often the problem. The GPIB address is 17.
        
     • (Obsolete) AGS-CNI-POLAR-FIBER2CAT is for the small black network box (which converts optical signals to copper signals) from which the ethernet cables are plugged to the other 3 units mentioned here.  This AC-cycle function here is virtually useless because if the ethernet has problem, you cannot use the ethernet to talk  to this and other 3 units anyway.  If the ethernet has no problem, you would not want to AC-cycle this unit.
        
     • AGS-CNI-POLAR-DVM is for the "Keithley" digital voltmeter.  On May 9, 2007,  AC-cycling this guy has somehow restored the communication for both the GPIB-ENET and DVM units via the ethernet.
        
     • AGS-CNI-POLAR-NETWORKSW  is for AC-resetting the p80F unmanaged Ethernet switch (which is of subnet 40/41) just downstream of the above "FIBER2CAT" unit.

      

   • If the bias currents of the 8 detectors don't update,
      
     • put DVM in  "CNI_HvBiasControl" to "offline" ,
        
     • wait for > 20 seconds or 1 minute ,
        
     • put DVM to "Remote" again.
       
        

   Actually, there is also the low voltage power supply ("Acopian" 6 & 12 V) for the pre-amplifier which one can't see from the TV monitor.  The node name of the WTI in AGS is called acr-c14-mod1.pbn.bnl.gov which you can ping to see whether it responds.

   The Hamamatsu detectors should be set at 80 V.
    

3.  Have the time for target insertion and retraction set correctly ?
   
   PolarControl is supposed to change the insertion and retraction time for each "Emittance measurement" as described below.   But after the emittance measurement is finished, it's supposed to put back the original value.  Occasionally PolarControl has failed to put back the original value.  So, one should check that the insertion and retraction time is correct !  Eg., if the DAQ window is fro 1.2 s to 1.85 s, and say the "target" needs 0.45 s to move (say, from park position of 95000 to 103000 --- where beam is), one may set the insertion time to be ~0.5 s so that it's stopped moving (at ~ 0.95 s) before the DAQ system starts taking data; the retraction time should be after 1.85 s such as 2.x s.
   
   
4.  How to change the RHIC toggle mode ?
   
   Goto  "pet" → "AGS" → "eXtraction" → double click "Tuning" → probably in the icon popped up at the bottom, on the line of "Spin mode", under "Control", you can click to switch between "Toggle" and "Fill mode".  Apparently, if ASA is present, "toggle" mode would generate AGS spin signals. We want it to be in the "Toggle" mode as our software polarControl checks on this.
   
   ASA is generated in "SuperMan". Under "View" in "SuperMan", you may click "Entire Table" and in Jan. 21, 2009, I saw that ASA was at 3 jiffies, the 2nd BT0 at 35 jiffies and AT0 at 36 jiffies → ASA is 32/60 seconds and 33/60 seconds before BT0 and AT0 respectively. The 1st BT0 was at 17 jiffies on that day which was only for conditioning of the Booster magnets to avoid any distortion by the later Booster cycle for NSRL in the same AGS cycle.
   
   Updated in 2014:  In recent years, we just use "Spin Orchestrator" to get into the "toggle" mode.
    
5.  Is attenuation set right ?
    
   • If I stopped an alpha calibration in the middle which needs attenuation of 5 without letting my script "entire_calib.csh" finish and restore the attenuation back to "1" (ie. no attenuation), no banana would be seen in the following measurements !
        
   • One such analysis file is shown here.
    
6.  BZDelay in the configuration file for the program "cnipol" probably may need to be adjusted every run, especially after the "rev-tick" (what I call "bunch-zero") signal is changed.  It has a range of [0..15].  It was set to be "3" in 2006 and in 2007 it needed to be changed to "12".   Changing this helps to move the "banana" to be close to the middle of the the curve of cut ranges.  If the value is too far off, the "banana" may be completely hidden.  ( To make the "banana" more perfectly in the middle of the cut ranges, calibration needs to be done. )
    
   • 1 BZDelay unit at 12^th Harmonics ~ 1/(70 MHz) ~ 14 ns; at 8^th Harmonics, it's ~21 ns.
      
   • One may also choose to change the "rev-tick" phase with respect to the wall current monitor signal (ie. the beam).  This may be done by going to "pet" → "Ags" → "Polarized_protons" → "RFparamsForPolarimeter" and change the 2^nd entry, namely "HITL 2 RF Sweep DDS Reset Phase (deg)".    360^o corresponds to one period of the"RF Sweep Reference Freq" in the same pet page which was set to ~4.45MHz in 2017, ie. ~225 ns.
     • The 1^st phase is "HITL 2 Revtick DDS  Reset Phase".  Its frequency "Rev Tick Reference Freq" (at the same pet page) was set ~370 kHz in 2017 and so each period is ~2.7 μs.  This can't change the banana curve in the time axis (except probably moving to other bunches).  Presumably because the rev-tick is a step-function/signal, it doesn't matter where the beam signal is inside step (range), until it go outside the step and moves to the next step.
     • For both cases, the range of degrees is from -180^o to 180^o.
     
        
   • Higher BZDelay values would move the bananas down (to smaller values) in the TOF (time-of-flight) axes.
      
   • To look for bananas at injection energy, it's especially difficult because the beam sizes are both considerably bigger longitudinally and transversely.  In 2013, we succeeded only after the cold snake was turned on which had the effect of reducing the transverse beam size (and seemingly also longitudinally as well).  The Gaussian σ of the longitudinal bunch length was ~20 ns.   The following plots show the "banana" discovery at different stages of BzDelay values : BzDelay=5,  BzDelay=8,  BzDelay=10 and BzDelay=11.
      
7.  Is the target aligned and in position (in the middle of the beampipe) for the beam to hit ?
   
   This may always be the problem and I've had it in my mind all the time.   It's difficult to know and fix the problem after the run has started because one needs an AGS access just to take a look at the target movement.  It's the best to be dealt with before the run has started.  It should be aligned as precisely as possible (by laser survey).
    
   •  The time specified for the target to go in and come out is crucial too.  In Jan. 2006, the lemo cables for the signal to control the target to go in and come out was swapped by mistake.  This gave us the illusion that retracting target earlier would have higher count rates !
      
     • To check how long and when the target stays in the beam, start "Gpm" → "Ags" → "Polarized_protons" → "CNItarget.mon" (you may need to click to show "All Monitor Files", NOT the "Log Request Files", to get the desired monitor).
       • In the Gpm page, if you want to add a device (to plot another parameter) and you are not sure about which ADO/FEC it should be though you know it's called "ael" something, you may do :
         
         cnsdump ADO | grep ael
         
         which would show :
         
         .........................
         ADO relMon aelMon.911b-vme.20 cfe-911b-vme 1000002 1 aelMon.agst0 n/911b
         .........................
         ADO relMon aelMon.f10-ps1.20 cfe-f10-ps1 1000002 1 aelMon.bak.agst0 n/f10
         .........................
         
         
         among others.   Here, the answer is "cfe-911b-vme" as I'm looking for "aelMon.agst0".

 
8.  Is the beam orbit at least good for C15 (where our target is) in AGS ?  

If the orbit goes away, it is difficult for the target to find the beam.   When there is nothing obvious, one may open the data collecting time window (the parameters START and STOP in the configuration file for the program "cnipol") to see whether one hits anything at all at any moment even before the flattop.
 

9.  Are the spin +/- signals arriving around the T0 of AGS ?
   
   The LeCroy 2367 seems to be detecting the spin +/- signals and the GCC starts counting soon after the parameter "RampBegin".  "RampBegin" can't be later than the dwelling time (before the ramping to the flattop); otherwise, the GCC that we obtain would be wrong.  Therefore, the spin +/- signals need to be available at the time of "RampBegin".
    
• Go to "pet"  → "Linac"  → "Oppis"  → "Spin", 2 panels pop up and look at the one at the bottom:
   
  • Under "AGS Event Link", one can change the pattern by putting cursor under "Pattern" and go to "Tools" at the top and click "Value Editor".   Change the spin pattern dimension by editing the "count" and put in 1 or -1 for spin + or -.
     
  • If one wants to change when the spin+/- signals should be generated, one changes the code under "Event(s)" to the code of the "Event" that one chooses.  At the time of writing, "AT0" is 20 and "ASA" is 236.  These codes can be checked on the EventLinkDisplay explained below.  In principle, one can also have more than one code in this software but I don't think we want that.
     
• After the afternoon on Feb. 9, 2006, O. Gould/L. Hoff has set the spin+/- signals to be generated at the time of the "ASA" (="AGS Spin Advance") event link signal by the RHIC Sequencer (one can set AGS Sequencer to the same using above "pet" page).  There are rules now to prevent "ASA" to go too far away from AGS and Booster T0's.  On Feb. 9, 2006, "ASA" was set to be ~500 ms before AGS T0. ( This seems to make sure that the spin signal is ready before AT0 and is the spin signal within that cycle --- marked by "SCS", the super-cycle. )
   
• To view the timing relationship, one can enter the command "EventLinkDisplay" in those acnxxx machines →  Click "AGS Event" on the right near the bottom
  → Click "Setup" ;
  → Click "Select Events" and select perhaps "AT0" (=AGS T0), "SCS" , "EV-SPIN-DOWN", "EV-SPIN-DOWN", and "ASA" ;
  → Click "Reset Display" and right-click anywhere inside the "text" page at the bottom and choose "Clear All" to clear everything and have a fresh start ;
  → Click "Acquire Once" for 1 cycle or "Acquire Continuous" for a few cycles and then click "Stop". 
  
  One could see an example here.  The "ASA" is about 516.7 ms earlier than "AT0". 
   
• The Booster signal "bij.pseudo_pkr.rt" (beam peaker) is about 2 ms after "AT0" and this is the signal which orders the beam.  The time of this beam peaker signal should be at least 300 ms after the spin signals are generated so that all the relevant signals have already arrived at the LINAC polarized proton source.  This way, we can be sure that the protons get the right spin sign requested by the RHIC Sequencer in the current super-cycle.
   
• Since Mar. 15, 2006 afternoon, with the help of O. Gould using a new V202 module, the spin+/- signals are now available in the entire AGS cycle.  Before by using a NIM delay/gate module, the spin+/- is only available for ~1.05 s.  So now, the spin+ signal would go down only when the spin- signal appears, and vice versa.
  
  There are 4 channels in this new V202 module: delay.hitl2SpinUpGateTrigger (channel 1), delay.hitl2SpinUpGate (channel 2), delay.hitl2SpinDownGateTrigger (channel 3) & delay.hitl2SpinDownGate (channel 4). To get access to them, go to "pet"  → "Tandem"  → "Location"  → "HITL_PS_HOUSE_2" → "cfe-hitl2-pol01".  The setup procedure is as follows:
   
  • Double-click "delay.hitl2SpinUpGateTrigger" to open up the "Dealy Channel Editor", click "Add Event" and add "EV-SPIN-UP" for all 4 PPM users.
     
  • With Double-click "delay.hitl2SpinUpGate" , click "Add Event" and add "EV-SPIN-Down" for all 4 PPM users.  After closing the panel, One has to set the mode to "Gate" instead of the pulse (default) for all 4 PPM users.
     
    • To set the mode to "Gate", close the "Delay Channel Editor, EITHER go to the top and click "Device" and click "Show All Parameters", OR click the middle mouse button, to pop up a new GUI panel.  Go down to the bottom and at "modeC", change "Pulse" to "Gate".  And do this for all 4 PPM users by clicking "PPM" at the top of this new panel.
       
  • Double-click "delay.hitl2SpinDownGateTrigger", click "Add Event" and add "EV-SPIN-DOWN" for all 4 PPM users.
     
  • Double-click "delay.hitl2SpinDownGate", click "Add Event" and add "EV-SPIN-UP" for all 4 PPM users.   One has to set the mode to "Gate" instead of the pulse (default) for all 4 PPM users, the same as for "delay.hitl2SpinUpGate".

  We actually make use of "delay.hitl2SpinUpGate" as the spin-up signal and delay.hitl2SpinDownGate as the spin-down signal.
  
  Diagnoses (!!):

  • If everything is working, at the same level as "modeC" above, one should see "pulseHistoryD" changed between "Yes" and "No" at the frequency of AGS cycle.
  • In the same place that one gets "delay.hitl2SpinDownGate" etc., one should find also "delay202Module.hitl2-pol01.B".   Click the middle mouse button and find "linkStatus".  It should show "OK" if the event links (ie., input) come into the V202 module alright.  If it shows "failed", it indicates that the event links (for us, the spin +/- signals) do not come in properly. 
    
    
10. Do we have more than one sequencer generating the spin +/- signals ?

If this is the case, the real spins of the protons may be unstable under 2 competing signals.  We've observed that when the Linac sequencer and AGS sequencer were both generating spin +/- signals during the 2007 run (behind RHIC stores of gold-gold), the signs of the polarizations may change each time when we mode-switched from "gold" to "polarized-proton".
 

• Linac sequencer: "pet"  → "Linac"  → "Oppis"  → "Spin", go to the bottom (one may need to enlarge the icon) to find "LEL.SEQNCR.1" which is the Linac sequencer switch.  "LEL.SEQNCR.1.C" works similarly as "AEL.SEQNCR.1.C" described above.
   
• Those signals with "LTI.RSPN_*" correspond to the RHIC Spin events.  Though they are not the RHIC sequencer, turning them off would turn off the respective signals (spin- up, down and zero) but turning them on without starting the RHIC sequencer would not turn on the actual RHIC Spin events.
   
• Those signals with "LOP.SPIN_*" are the actual events that are sent to the polarized-proton source.  These are the "OR's" of all signals from RHIC, AGS and Linac sequencers.
   
11.  Are the proton spins flipping from the source ?
    
    The place to set and check the laser triggering so that the proton spins would be flipping up and down is:  "pet"  → "Linac"  → "Oppis"  → "CDC.OPPIS.V202B"  → "LOP.LASER_TRG.ST"  → under the column "C1", set it to "ON" (for the right PPM user, eg. "user 4"). 
    
    The pulses of protons sent to the 200 MeV polarimeter (at the end of LINAC) are different from the pulses of protons sent to Booster/AGS.  In 2015 before Jan. 5 ~11 am, the above trigger was turned on for user 5 (for 200 MeV polarimeter) but not for user 4.  Consequently, though the 200 MeV polarimeter measured >70% polarization whereas the AGS CNI polarimeter has measured ~0%.
     
    
    
12.  Beam intensity (AIS.AGS_EXT.CBM) not in sync with our DAQ number-of-event measurement ?
    
    The agn.cmb_read.rt was set after "AGE" (AGS Group End) which is too late.  It was OK for users 1 and 4 as there was another ada.end_seb_spill which occured earlier and made it OK.  But in 2007, we used the PPM user 3 and we found that the AIS.AGS_EXT.CBM signal came in 1 cycle late.  This problem was fixed (moving agn.cmb_read.rt earlier and deleting ada.end_seb_spill).

The sequences of things that need to happen are:
 

• Pulses of AIS.AGS_A_TRANS.CBM, AIS.AGS_B_TRANS.CBM, AIS.AGS_EARLY_TRANS.CBM, AIS.AGS_EXT.CBM and AIS.AGS_INJ.CBM need to come to the scalers. 
   
• These scaler readings need to be shadowed (copied into registers).  This hardware operation probably takes only some μs.
   
• Software reads into VME memory from the registers at the time of "AGE". The timing of "AGE" is managed by "SuperMan" and it is ~0.5 second before the next "AT0".
   
• Software generates the reports at AGE + 100 ms.

 The timing of last two actions is set by the front-end configuration and does not normally change.

 To check/set the shadowing operation, go to "pet" → "Location" → "Ags" → "911B" → "CDC.911SWIC.V202A" → look for Device Name "ADL.ACTSIS.SDW.CLK" ("ACT" is the AGS Current Transformer, "SIS" is the scaler card and "SDW" stands for the "shadower") → double left-click and you'd see "agn.cmb_read.rt" under the "Trigger".

One should to set/check this for all 4 PPM users.

To set the time for  "agn.cmb_read.rt", "pet" → "AGS" → "Timing" → "ags_RT_gen" and look for "agn.cmb_read.rt".  It is now set at 2850000 μs (=2.85 s) after "AT0".  To change it, middle-click on "agn.cmb_read.rt" → put the cursor on the value next to "clockCountS" → seeing that it's like [2850000] indicates that it's an array → goto top menu and click "Tools" and select "Value Editor".  There you can change the value itselt (or even the array length but we probably don't want to do so).

At the time of writing and as an example, the supercycle is ~4 s and "AGE" happens at ~3.5 s ⇒ there is a duration of ~(3.5-2.85) = 0.65 s before the software read at "AGE" happens.  If the supercycle is much shorter ( like ~3 s, say), we'll need to change when   "agn.cmb_read.rt" is to occur.

To look at AIS.AGS_EXT.CBM from the pet page, "pet" → "Instrumentation" → "Current_xfmrs" → "Ring_xf".

 

13.  Bananas disappear due to huge event rates for pulses ?
    
    In Feb. 2009, banana curves might suddenly disappear (such as shown here). I have observed that the symptom was that the event rates were very high (often several millions) as compared to normal event rates of 500K or 600K, for at least for some pulses during a 40 M event run. One huge pulse might be enough to screw up the banana. 

    It's been puzzled until Feb. 13 when it happened again, Kevin Brown was able to "fix" this problem by removing a useless radial shift near ~600 ms (which was even my data-taking windows started at ~1 s) --- the black lines were before the removal and the red lines were after removal.  Before this removal, the average beam orbit looked like this; after removing the radial shift, the average beam orbit looked like this.

     
14.  Huge signals even without beam ?
    
    If one sees huge signals even without beam, some channels/strips in the silicon detectors may be bad/damaged.  This probably has happened when there was beam loss nearby or beam hitting the target frame accidentally, the silicon detectors got badly hit.
     
    • Our solution at one point was just to take away those 2 channels (simply unplugging the lemo cables from the WFD's), the rest of the channels would readily work (giving proper banana curves and carbon mass peaks).
    • Somehow, the damage of the above-mentioned 2 channels were not permanent.  After we didn't use them for a few days and went into the AGS ring to reset the bias voltage ( and might or might not have replaced the pre-amplifier ), the 2 channels came back alive.
    • I've also observed that after the read-out rate got very high, the rate would decrease gradually over time if it was not used.
       
15.  Is AGS-RHIC RF synchro off for us (especially when RHIC synchro is undefined when RHIC is off) ?   

    If this RF synchro is on, the orbit may go away or crazy.  On Jan. 3/4 (2012) midnight, the beam was debunched at flattop as the synchro was on when RHIC was off and synchro was undefined. It showed symptoms like when RF were off at flattop.

    [ Obsolete after RF upgrade in 2013: To check/turn off the RF synchro, start "pet" → "Ags" → "Rf" → "rf_Beam_control" → "Feedback_loops_beam" →  choose the right "PPM User" → On the line with "ARF.SYNCH_LOOP.ST", make sure that it is "OFF" under "C1".  "ON" means RF synchro is on.]

    To turn off "green synchro" in the new RF system ≥ 2013 (as Keith Zeno taught me),  start "pet" → "Ags" → "extraction"  → "Tuning" → set both "ATR synchro hold loops+AC loop" and "ATR Synchro Phase Loop Start" to "off".
    
    In 2013 ( at least ), we would see "hairy" structure in profile plot from the emittance measurement such as here when RF synchro was ON, instead of the "green synchro". Changing to "green synchro" would solve the problem.

    In 2014, Keith Zeno told me to check: "pet" → "AGS" → "Rf" → "AGS" → "cfe-929-rfll1" → "Timing" → "delayV202Dsp" → the "Enable" column on the row of "Loop Hold for AtR Synchro"    to see whether it's "Off".

16.  Is the RF turned on at AGS flattop ?

    For the CNI polarimeter, RF needs to be turned on but it needs to be off for the E880 polarimeter.  So, watch the TV in (eg.) HITL2  !

    If RF is off, we wouldn't be able to see carbon mass peaks and banana curves.  And the bunch no. distribution may look like this.
    
    Since Andrei's new analysis program (at least as of Spring 2013) doesn't have a check or cut against certain % of events have to be inside the banana bands, when there is ~uniform distribution of particles (ie. no banana) in the time-of-flight and energy plot, the polarization would be reported as something about 30% or 40%.
     

17.  Is AGS CDEV server for us working ?

     I have a little but powerful program in ~kinyip/cni2005-dev/cnipol/cdevTest (which tries to set the run no.) to test whether CDEV is working.  To reset our specialized AGS CDEV server, (as user "mcr",) start "https://startup.pbn.bnl.gov/servers"
                                               → "mcr_servers..."
                                               → "AGSPolarServer (Proton Run)"
                                               → "restart" (  one may "check" first )

18.  Check the beam loss monitor at C15 (though probably many people may already be checking).   If you see any substantial "peaks" in radiation, it may be a sign of the beam hitting the target frame or something worse.
     
19.  Xilinx FPGA's in WFD is malfunctioning ?
    
    One time (2006), I have seen the following plots for the banana curve and -t distribution.  The solution was to reprogram the Xilinx FPGA's in the WFD's.
    
    
    
20.  Target problem ?
    
    If the target refuses to move or is just out of control, one may hit the button hidden below "Home" called "Here" which is essentailly a "reset".
    
    Sometimes when "polarControl" get no event, it is because the target doesn't move.  It happened in the evening of Feb. 2.  Though the horizontal targets could move, the vertical ones couldn't or moved intermittently. At the end, we (D. Gassner) went to HITL2 and disconnected/re-connected the timing signal cables connecting V202 to the V (~vertical) VME module { and H (~horizontal) VME module but this shouldn't matter as the horizontal targets had been working anyway }.
     
21.  Banana curves move in the TOF axis ?  Rev Tick (=bunch-zero signal) dependence on beam intensities ...
     
    In the 2007 run, after a long time, I'd finally realized that the "rev tick" (our bunch-zero signal) was sensitive to the beam intensities.  We saw the same behavior during the beginning of this run (2008).  That is, the banana curves would move up and down in the TOF (time-of-flight) axis when the intensities changed.
     
    After the 2007 run, I had started to push the RF group (mainly Freddy) to use higher harmonics to generate the "rev tick" --- only because before Joe de Long left BNL, he was doing that to generate rev. tick.  On Jan. 31, 2008 (Thursday), this was finally materialized when Freddy finally got around to do it.  And after a few days' observation, it definitely seemed that the "rev tick" was no longer sensitive to the beam intensities.
     
    Freddy actually has used the 4th harmonics to generate the "rev tick". 1st harmonics means the revolution frequency in AGS. 4th harmonics means 4 times the revolution frequency.
     
22.  Emittance measurement problems
    
    
    • To have the emittance measurement working, the target insertion/retraction times are crucial !  One has to go to  "pet"  → "AGS"  → "Polarized-protons"  → "PolarControlDefaults"and click "Summary".  During the Emittance measurements, polarControl would remember what the original insertion/retraction times are and then replace them with what are set in the "PolarControlDefaults" page. After measurements, polarControl would put the original insertion/retraction times back. Eg. When the "Start" and "Stop" times for the beginning and end of data acquisition are 1100 and 1800 respectively (in the configuration file ~mcr/ado/new_set.ini), I could set "insertion/retraction" times to be 1.1 s and 1.82 s respectively. [ The normal insertion time for "target measurement" is "0.7 s" ]
       
      • Be careful to choose the appropriate PPM user in both the "polarControl" application and "PolarControlDefaults".
         
    • In the first year of commissioning in Feb. 2008, the main problem was to get the target to move into the beam once and only once.  At the end, we understood that the target would only move if the command to ask it to move (eg. "Target1") is before the insertion time (eg. 1.2 s that we'v set) with respect to AT0.  The simple solution at the end is that we send "Target1" (say) after we receive AT0 (code 20) or whatever the trigger signal set in the delay module { eg.  AT4 (code 24) --- AT0 copied for User 4).  And then we wait for the next AT0 before we send the command to "RotateOut".  This way seems to guarantee that the target would enter the beam only once.

    In one commissioning version of polarControl, I asked Seth to specifically to let my DAQ system start many seconds before the target went in and wait many seconds after the target moved out.   Somehow, this way, the histogram (71 in the hbook file) would accumulate a lot of entries at the encoding position ~ 0.   This somehow screwed up the intelligence of my fitting routine fit.C which "smartly" tries to use the bin with the largest no. of entries as the candidate for the peak position of the Gaussian fit.  My simple solution is to make the lower edge of the histogram 71 to start from 10, rather than 0.

    Here is a Gpm plot which shows that the target moves to the "start" position and wait for a couple cycles (it's actually 5 s and then wait for the next beginning of the cycle) before it moves to the "end" position.

23.  The problem with minimum momenta or Gγ values ( GCC = 0 )
    

    In the 2009 run,  I saw non-zero entries with minimum/lowest end of  momentum or Gγ spectrum in the histograms of the first page of the "expert" plots such as this.  After a while, I finally was smart enough to check and find that they all came from the 4 channels from a WFD and the GCC were always 0 for those 4 channels. Each WFD has one GCC input. Going to HITL2 and checking the GCC connection, I found the loose lemo (almost unplugged actually) near the GCC fan-out NIM module.  Plugging the lemo back has solved the problem.

    Of course, one has to make sure that the "RampBegin" parameter in the configuratio file "new_set.ini" is at about the place where the AGS ramp begins (~160 ns).  Because "RampBegin" is the time when the WFD starts to count GCC.  Setting "RampBegin" too late would prevent WFD from catching the earlier GCC signals and therefore the maximum momentum/Gγ would be smaller than what they should be.

    
    

24.  User 4 → User 2 (late Feb. 2008):
    
    
    • At first, the target didn't move when I clicked the emittance measurement "start" button in the "polarControl" application. The next day, Seth said that it was because fecSystem.hitl2-pol01 was not changed to User 2. A. Marusic told me to check by the command

      adoIfA -T fecSystem.hitl2-pol01 activePpmUsers

      When I checked (the next day), fecSystem.hitl2-pol01 did indicate "User 2" and the target indeed moved. Hmm ... so I couldn't repeat the failure that I saw. But at least, this was the place to check and actually, "fecSystem.hitl2-pol01:activePpmUsers" was at the bottom of the "CNI_Target" pet page.
      
      
    • The 1st problem was that we didn't get the intensity, ie. AIS.AGS_EXT.CBM always returned 0. The solution was to follow above to turn on "agn.cmb_read.rt" for User 2 and set it to be 2.85 s after AT0.
      
      
    • The 2nd problem was that the insertion/retraction times was not set at all because polarControl was using the default user which was 4 and adjusted the insertion/retraction times for the User 4. But the target was controlled by User 2. The solution was to set polarControl to use the "currentPpmUserM" (which was 2, the correct one to look at).
       
      
      
25.  The problem with the frequency multiplier:
    
    
    • These two banana plots for the 90° detectors, asym_42747_p9.ps and asym_42749_p9.ps, show the situation when this problem appeared.  The situation in one was somewhat better than the other.  The banana plots for the 45° detectors were OK.
      
      
    • The clock signals to the WFD's for the 90° and those for the 45° come from 2 separate frequency multipliers. When I checked the (output) clock signals from the two frequency multipliers, they look like: with 50 Ω termination or AC coupled.  It's difficult to tell which one is good or which one is bad.
      
      
    • At the end, I swapped those bunch-zero, clock and inhibit signals (since I didn't know which of the 3 was the culprit) between one WFD's for 90° and another one with 45°.  I expected to find nothing (because the signals looked the same to me in the oscilloscope). But, after a few trials, I could conclude that changing the clock signals immediately made the 90° show bananas for those (4) channels (within that WFD) and made 45° bananas disappear.
       
    • Something must have been marginal. In fact, at some point, by replugging the clock signals to the unused slot on the "bad"/marginal freq. multiplier, all the bananas showed up perfectly.  But either what I jotted down were inaccurate or it went bad again (as things were marginal), when I tried ~10 times (~10 combinations) later, nothing was as perfect as that one single time and was unusable.  So, at the end, I swapped the "problematic" frequency multiplier with another one.
       
       
26.  Wierd GCC spectrum and polarization instability
    

    On Feb. 20, 2010, we started to see GCC plot such as asym_46212_p1.pdf, ie., it has a continuous spectrum at low GCC values instead of a unique bin at one high value.  This was accompanied with big χ² (>>1) indicating large fluctuations in polarizations/asymmetries.  Cutting away the lower GCC values would get rid of the large fluctuations in asymmetries and χ². At the end, this was found to be due to the loosened ECL cable that was/is plugged  to the LRS2367. After pushing it in all the way, the above problem has then disappeared.

     

27.  Changing the CC32 cable could cause problem
    

    Apparently on Dec. 20, 2010, I unplugged and plugged back in the CC32 cables between the PC (cnipol2) and the CAMAC crate where LRS2367 was.  Then, the emittance plots were found to be emtpy or half Gaussian (or occassionally successful) and you'd see some cycles getting 10 times more events (like millions).  At the end, after I rebooted the PC, the problem went away.  Very likely, the CC32 driver (in Linux) is not plug-&-play and unplugging & plugging back the CC32 cable caused the weird problem.

 

28.  How to watch C-A TV (live) online ?
     
    [ Obsolete ! ] Click the "C-A Broadcast page"  → go to the top and click on the "small" scroll bar to go down to the bottom of that tiny page (very obscure !!) → click on "C-A TV"  → click the link at the top to watch the livefeed.ram (with realplayer etc.).

 

29.  Early or Late CBM or Booster_Input all are zeroes in the Krisch sheet ?
     
    Soultion: Restart "Krisch Server" (from https://startup.pbn.bnl.gov/servers ) for the right ppm-user (ie. when the proton beam is operated in AGS user 4, you probably don't want to do this when AGS is in user 1).

 

30.  Function Generator Issues
    
    Starting in 2014, we made use of the clock counts from John Morri's "Function Generator" in place of GCC during ramp measurements.  (The clock counts appear in channel 4 of the MUX_DAQ on the pet page.  GCC is in channel 3.)  
    
    [ Obsolete since 2021.  See the 2021 instructions instead !
    
    To start it, one may use "StartUp" to start the "Commissioning" version (Pol Timing) of the FunctionEditor
    → double-click on "AGS" → select "PolarizedProtons" → select "Timing" on the right → clock "OK" at the bottom
    → under "File", choose "Open"/"Library" → pick "150msTo650ms_50KHz" → click "Make Live" to send the function.
    
    To clean it (to save memory as there may not be enough cache memory for all 4 users.  When that happened, the function may not get sent next time when somebody reboots the cfe-htil2-pol chassis as insufficient cache memory doesn't contain that bit of information for the user that we care such as "user 4"),  do almost the same as above except that we pick "zeroFunc" to send. ]
    
    Diagnosis : 
    
    On the right hand side of the "FunctionEditor", click the "Pet Page" button and, put  your cursor on any of the "qfg.agspol-time" items.  Then, under "Device" at the top, click "Show Parameters".
    
    
    • Under "eventFormulaS", it should be "20" (as AGS T0 for all users).  In 2014, after a reboot by Al Marusic, as there wasn't enough cache memory, "20" wasn't restored for "eventFormulaS" and we got no clock count signals from this Function Generator. After one sets "eventFormulaS" to "20", one needs to click "Make Live" to make it take effect !
      
      
    • Another place to watch is "linkErrorM" which should be "None" when there is no problem.  "stateS", "stateM" and "statusM" should be "empty" and "statusM" is zero ("0x0").
      
    
    [ Obsolete since 2021.  See the 2021 instructions instead !
    
    New choice in 2017: 
    
    Instead of "150msTo650ms_50KHz", J. Morris provided a new function "50KHz1msecForRepeat_Start" which repeats the "1 ms pattern" 500 times (instead of one "500 ms pattern"). 
    
    
    • Usage of it is similar to "150msTo650ms_50KHz" but in addition, after loading "50KHz1msecForRepeat_Start" from the library, you need to click the "pet" and set "repeatCount" to be 500 (not 1 !). 
    ]
    
    In 2021: 
    
    Now, we go to the Pet page, select "AGS" → select "PolarizedProtons" → select "TriggersForRampMeas", at the top line that one may change settings,
    → put in "150" for the "start time (ms)" and put in "500" for "duration (ms)",
    
    after hitting return, you'll see (on the line below) :
    → "start time (s)" as "0.15001,  "repeat count" changing to "500" and "last time (s)" changing to "0.65".   At 50 kHz, this will give us 25000 counts.
    
    


31. Bunch hopping
     
    In the evening of May 31, 2015, I was called by MCR and we've finally realized that there was a problem of bunch hopping.  If I analyzed the events in the correct bunch (bunch 2 in my naming scheme), the banana plot seems fine.   But if I analyzed the events in the wrong bunch (bunch 22 in my nameing scheme), the plot contains background/prompt events.
    
    When I looked at the log file 67464.log, I saw:
    
    4 SP:+ 0 ( 9927) EV:   7167613 ( 919) 7799.36 TOT: 7.168*10^6
    4 SP:+ 1 ( 9929) EV:   6763603 ( 984) 6873.58 TOT: 13.931*10^6
    4 SP:+ 2 ( 9930) EV:     502382 ( 939) 535.02 TOT: 14.434*10^6
    4 SP:- 3 ( 9932) EV:   7172725 ( 949) 7558.19 TOT: 21.606*10^6
    4 SP:- 4 ( 9933) EV:     561891 ( 1049) 535.64 TOT: 22.168*10^6
    4 SP:+ 5 ( 9934) EV:    544142 ( 1032) 527.27 TOT: 22.712*10^6
    4 SP:- 6 ( 9936) EV:   6847385 ( 989) 6923.54 TOT: 29.560*10^6
    4 SP:- 7 ( 9938) EV:   7101107 ( 909) 7812.00 TOT: 36.661*10^6
    4 SP:- 8 ( 9939) EV:     550151 ( 1017) 540.95 TOT: 37.211*10^6
    4 SP:+ 9 ( 9940) EV:    534318 ( 972) 549.71 TOT: 37.745*10^6
    4 SP:- 10 ( 9942) EV: 7404133 ( 907) 8163.32 TOT: 45.149*10^6
    ...
    
    One'd see that when the bunch was in the wrong place, a lot of background/prompt events were collected which were ~10 times of the usual no. of events.  But when it was in the right bunch, only hundred of thousands of events were received.   When this problem happened, the intensities from cycle to cycle also fluctuated more than usual.
     
32.  Emittance measurement shows "empty page" because the steps are always 0 ?
    
    On Dec. 14, 2016, when I started using polarimeter after almost 2 years, I found that the emittance measurement result was blank.  I checked the run.root in ~mcr/ado and found that there were many entries in histogram "h71" but all of them were "underflow", which (most likely) means that they were all "0".
    
    Peter Zimmerman and I went to HITL2 and we found that the MUX crate was turned on.  Peter checked and found that the -6V read ~0 though +6V read correctly.  So, we just turned off the power module in the NIM crate and then on.  And voila, when I tried the emittance measurement again, I saw the ~Gaussian curve !
    
    Afterwards, I've checked that when it works, all ±6, ±12 and ±24 volts for that power supply module should be on.
    
    

33. Emittance blowup at flattop ?
    
    About 3 am on Feb. 3, 2017, MCR called me because they found that the 3rd picture on the left of this monitoring page (the grey area) showed decreasing no. of events sometime after 1.6 s (after AT0).  At the end, I've found that it's due to the emittance blowup after 1.6 s (after AT0) at flattop as shown in this IPM's emittance measurement plot (the left one at the bottom is the relevant one for the horizontal emittance when we were using vertical target 1).




34. WinShift needs adjustment (noise vs χ² stability)
    
    In the 2022 run, the polarization χ² 's (variation) were suddenly big (20, 30 or larger, instead of 1) --- which would make the polarization values unreliable due to its huge fluctuation.  The machine experts couldn't find anything wrong and changing pre-amp and targets didn't solve the problem (completely).  At the end, it was found that if I changed the WinShift (in the "new_set.ini" file) from 3 to 11 (setting considerably higher than 11 like 15 would see huge noise), the χ²  instability went away.
    
    But later, the RF signals jumped again and I needed to re-adjust WinShift to smaller values.