Must-do checklist (that one might quite easily forget !) :


What could go wrong during commissioning or even later ?

  1. A hung "cnipol" process is preventing further polarimeter action

    For some unknown reason, once a while, a hung process in the polarimeter PC (cnipol3 ~ 2015) may refuse to stop/finish.  So, "ssh" to "cnipol3" and enter the following :

    ps -ef | grep cnipol

    You may see something like ("cnipol_noprog" in the example below may be just "cnipol") :

    mcr  27095 27091  0   08:25   ?   00:00:00 /home/mcr/ado/cnipol/cnipol_noprog -m -n65578 -d001 -u4 -e40000000

    If you found the above when nobody was taking any measurement, you should kill the process by doing:

    kill 27095

    where "27095" comes from the first no. in the output line of "ps -ef | grep cnipol".


  2. How about the bias voltage for the silicon detector ?

    Check the bias voltage and current.  At the time of writing (early 2006), it's by looking at the TV monitor in HITL2 (Bldg. 908).  If it trips, it'll be zeroes all over the places and the CNI polarimeter system certainly wouldn't work if the bias voltage trips.  Here is a set of analysis plots that was made for a run when the bias voltage tripped.

    Starting from the 2007 run,  one can turn on the bias voltage from "pet"  → "AGS"  → "Polarized-protons"  → "CNI_HvBiasControl".   I've also observed that after the read-out rate got very high, the rate would decrease gradually over time if it was not used.

    Actually, there is also the low voltage power supply ("Acopian" 6 & 12 V) for the pre-amplifier which one can't see from the TV monitor.  The node name of the WTI in AGS is called acr-c14-mod1.pbn.bnl.gov which you can ping to see whether it responds.

    The Hamamatsu detectors should be set at 80 V.
     

  3.  Have the time for target insertion and retraction set correctly ?

    PolarControl is supposed to change the insertion and retraction time for each "Emittance measurement" as described below.   But after the emittance measurement is finished, it's supposed to put back the original value.  Occasionally PolarControl has failed to put back the original value.  So, one should check that the insertion and retraction time is correct !  Eg., if the DAQ window is fro 1.2 s to 1.85 s, and say the "target" needs 0.45 s to move (say, from park position of 95000 to 103000 --- where beam is), one may set the insertion time to be ~0.5 s so that it's stopped moving (at ~ 0.95 s) before the DAQ system starts taking data; the retraction time should be after 1.85 s such as 2.x s.

  4.  How to change the RHIC toggle mode ?

    Goto  "pet" → "AGS" → "eXtraction" → double click "Tuning" → probably in the icon popped up at the bottom, on the line of "Spin mode", under "Control", you can click to switch between "Toggle" and "Fill mode".  Apparently, if ASA is present, "toggle" mode would generate AGS spin signals. We want it to be in the "Toggle" mode as our software polarControl checks on this.

    ASA is generated in "SuperMan". Under "View" in "SuperMan", you may click "Entire Table" and in Jan. 21, 2009, I saw that ASA was at 3 jiffies, the 2nd BT0 at 35 jiffies and AT0 at 36 jiffies → ASA is 32/60 seconds and 33/60 seconds before BT0 and AT0 respectively. The 1st BT0 was at 17 jiffies on that day which was only for conditioning of the Booster magnets to avoid any distortion by the later Booster cycle for NSRL in the same AGS cycle.

    Updated in 2014:  In recent years, we just use "Spin Orchestrator" to get into the "toggle" mode.
     
  5.  Is attenuation set right ?
       
  6.  BZDelay in the configuration file for the program "cnipol" probably may need to be adjusted every run, especially after the "rev-tick" (what I call "bunch-zero") signal is changed.  It has a range of [0..15].  It was set to be "3" in 2006 and in 2007 it needed to be changed to "12".   Changing this helps to move the "banana" to be close to the middle of the the curve of cut ranges.  If the value is too far off, the "banana" may be completely hidden.  ( To make the "banana" more perfectly in the middle of the cut ranges, calibration needs to be done. )
     
  7.  Is the target aligned and in position (in the middle of the beampipe) for the beam to hit ?

    This may always be the problem and I've had it in my mind all the time.   It's difficult to know and fix the problem after the run has started because one needs an AGS access just to take a look at the target movement.  It's the best to be dealt with before the run has started.  It should be aligned as precisely as possible (by laser survey).
     

  8.  
  9.  Is the beam orbit at least good for C15 (where our target is) in AGS ?  
  10. If the orbit goes away, it is difficult for the target to find the beam.   When there is nothing obvious, one may open the data collecting time window (the parameters START and STOP in the configuration file for the program "cnipol") to see whether one hits anything at all at any moment even before the flattop.
     

  11.  Are the spin +/- signals arriving around the T0 of AGS ?

    The LeCroy 2367 seems to be detecting the spin +/- signals and the GCC starts counting soon after the parameter "RampBegin".  "RampBegin" can't be later than the dwelling time (before the ramping to the flattop); otherwise, the GCC that we obtain would be wrong.  Therefore, the spin +/- signals need to be available at the time of "RampBegin".
     
  12. Do we have more than one sequencer generating the spin +/- signals ?
  13. If this is the case, the real spins of the protons may be unstable under 2 competing signals.  We've observed that when the Linac sequencer and AGS sequencer were both generating spin +/- signals during the 2007 run (behind RHIC stores of gold-gold), the signs of the polarizations may change each time when we mode-switched from "gold" to "polarized-proton".
     

  14.  Are the proton spins flipping from the source ?

    The place to set and check the laser triggering so that the proton spins would be flipping up and down is:  "pet"  → "Linac"  → "Oppis"  → "CDC.OPPIS.V202B"  → "LOP.LASER_TRG.ST"  → under the column "C1", set it to "ON" (for the right PPM user, eg. "user 4"). 

    The pulses of protons sent to the 200 MeV polarimeter (at the end of LINAC) are different from the pulses of protons sent to Booster/AGS.  In 2015 before Jan. 5 ~11 am, the above trigger was turned on for user 5 (for 200 MeV polarimeter) but not for user 4.  Consequently, though the 200 MeV polarimeter measured >70% polarization whereas the AGS CNI polarimeter has measured ~0%.
     

  15.  Beam intensity (AIS.AGS_EXT.CBM) not in sync with our DAQ number-of-event measurement ?

    The agn.cmb_read.rt was set after "AGE" (AGS Group End) which is too late.  It was OK for users 1 and 4 as there was another ada.end_seb_spill which occured earlier and made it OK.  But in 2007, we used the PPM user 3 and we found that the AIS.AGS_EXT.CBM signal came in 1 cycle late.  This problem was fixed (moving agn.cmb_read.rt earlier and deleting ada.end_seb_spill).
  16. The sequences of things that need to happen are:
     

     The timing of last two actions is set by the front-end configuration and does not normally change.

     To check/set the shadowing operation, go to "pet" → "Location" → "Ags" → "911B" → "CDC.911SWIC.V202A" → look for Device Name "ADL.ACTSIS.SDW.CLK" ("ACT" is the AGS Current Transformer, "SIS" is the scaler card and "SDW" stands for the "shadower") → double left-click and you'd see "agn.cmb_read.rt" under the "Trigger".

    One should to set/check this for all 4 PPM users.

    To set the time for  "agn.cmb_read.rt", "pet" → "AGS" → "Timing" → "ags_RT_gen" and look for "agn.cmb_read.rt".  It is now set at 2850000 μs (=2.85 s) after "AT0".  To change it, middle-click on "agn.cmb_read.rt" → put the cursor on the value next to "clockCountS" → seeing that it's like [2850000] indicates that it's an array → goto top menu and click "Tools" and select "Value Editor".  There you can change the value itselt (or even the array length but we probably don't want to do so).

    At the time of writing and as an example, the supercycle is ~4 s and "AGE" happens at ~3.5 s ⇒ there is a duration of ~(3.5-2.85) = 0.65 s before the software read at "AGE" happens.  If the supercycle is much shorter ( like ~3 s, say), we'll need to change when   "agn.cmb_read.rt" is to occur.

    To look at AIS.AGS_EXT.CBM from the pet page, "pet" → "Instrumentation" → "Current_xfmrs" → "Ring_xf".

     

  17.  Bananas disappear due to huge event rates for pulses ?

    In Feb. 2009, banana curves might suddenly disappear (such as shown here). I have observed that the symptom was that the event rates were very high (often several millions) as compared to normal event rates of 500K or 600K, for at least for some pulses during a 40 M event run. One huge pulse might be enough to screw up the banana. 

    It's been puzzled until Feb. 13 when it happened again, Kevin Brown was able to "fix" this problem by removing a useless radial shift near ~600 ms (which was even my data-taking windows started at ~1 s) --- the black lines were before the removal and the red lines were after removal.  Before this removal, the average beam orbit looked like this; after removing the radial shift, the average beam orbit looked like this.

     
  18.  Huge signals even without beam ?

    If one sees huge signals even without beam, some channels/strips in the silicon detectors may be bad/damaged.  This probably has happened when there was beam loss nearby or beam hitting the target frame accidentally, the silicon detectors got badly hit.
     
  19.  Is AGS-RHIC RF synchro off for us (especially when RHIC synchro is undefined when RHIC is off) ?   

    If this RF synchro is on, the orbit may go away or crazy.  On Jan. 3/4 (2012) midnight, the beam was debunched at flattop as the synchro was on when RHIC was off and synchro was undefined. It showed symptoms like when RF were off at flattop.

    [ Obsolete after RF upgrade in 2013: To check/turn off the RF synchro, start "pet" → "Ags" → "Rf" → "rf_Beam_control" → "Feedback_loops_beam" →  choose the right "PPM User" → On the line with "ARF.SYNCH_LOOP.ST", make sure that it is "OFF" under "C1".  "ON" means RF synchro is on.]

    To turn off "green synchro" in the new RF system ≥ 2013 (as Keith Zeno taught me),  start "pet" → "Ags" → "extraction"  → "Tuning" → set both "ATR synchro hold loops+AC loop" and "ATR Synchro Phase Loop Start" to "off".

    In 2013 ( at least ), we would see "hairy" structure in profile plot from the emittance measurement such as here when RF synchro was ON, instead of the "green synchro". Changing to "green synchro" would solve the problem.

    In 2014, Keith Zeno told me to check: "pet" → "AGS" → "Rf" → "AGS" → "cfe-929-rfll1" → "Timing" → "delayV202Dsp" → the "Enable" column on the row of "Loop Hold for AtR Synchro"    to see whether it's "Off".

  20.  Is the RF turned on at AGS flattop ?

    For the CNI polarimeter, RF needs to be turned on but it needs to be off for the E880 polarimeter.  So, watch the TV in (eg.) HITL2  !

    If RF is off, we wouldn't be able to see carbon mass peaks and banana curves.  And the bunch no. distribution may look like this.

    Since Andrei's new analysis program (at least as of Spring 2013) doesn't have a check or cut against certain % of events have to be inside the banana bands, when there is ~uniform distribution of particles (ie. no banana) in the time-of-flight and energy plot, the polarization would be reported as something about 30% or 40%.
     

  21.  Is AGS CDEV server for us working ?

     I have a little but powerful program in ~kinyip/cni2005-dev/cnipol/cdevTest (which tries to set the run no.) to test whether CDEV is working.  To reset our specialized AGS CDEV server, (as user "mcr",) start "https://startup.pbn.bnl.gov/servers"
                                               → "mcr_servers..."
                                               → "AGSPolarServer (Proton Run)"
                                               → "restart"one may "check" first )

  22.  Check the beam loss monitor at C15 (though probably many people may already be checking).   If you see any substantial "peaks" in radiation, it may be a sign of the beam hitting the target frame or something worse.
     
  23.  Xilinx FPGA's in WFD is malfunctioning ?

    One time (2006), I have seen the following plots for the banana curve and -t distribution.  The solution was to reprogram the Xilinx FPGA's in the WFD's.


  24.  Target problem ?

    If the target refuses to move or is just out of control, one may hit the button hidden below "Home" called "Here" which is essentailly a "reset".

    Sometimes when "polarControl" get no event, it is because the target doesn't move.  It happened in the evening of Feb. 2.  Though the horizontal targets could move, the vertical ones couldn't or moved intermittently. At the end, we (D. Gassner) went to HITL2 and disconnected/re-connected the timing signal cables connecting V202 to the V (~vertical) VME module { and H (~horizontal) VME module but this shouldn't matter as the horizontal targets had been working anyway }.
     
  25.  Banana curves move in the TOF axis ?  Rev Tick (=bunch-zero signal) dependence on beam intensities ...
     
    In the 2007 run, after a long time, I'd finally realized that the "rev tick" (our bunch-zero signal) was sensitive to the beam intensities.  We saw the same behavior during the beginning of this run (2008).  That is, the banana curves would move up and down in the TOF (time-of-flight) axis when the intensities changed.
     
    After the 2007 run, I had started to push the RF group (mainly Freddy) to use higher harmonics to generate the "rev tick" --- only because before Joe de Long left BNL, he was doing that to generate rev. tick.  On Jan. 31, 2008 (Thursday), this was finally materialized when Freddy finally got around to do it.  And after a few days' observation, it definitely seemed that the "rev tick" was no longer sensitive to the beam intensities.
     
    Freddy actually has used the 4th harmonics to generate the "rev tick". 1st harmonics means the revolution frequency in AGS. 4th harmonics means 4 times the revolution frequency.
     
  26.  Emittance measurement problems

    • In the first year of commissioning in Feb. 2008, the main problem was to get the target to move into the beam once and only once.  At the end, we understood that the target would only move if the command to ask it to move (eg. "Target1") is before the insertion time (eg. 1.2 s that we'v set) with respect to AT0.  The simple solution at the end is that we send "Target1" (say) after we receive AT0 (code 20) or whatever the trigger signal set in the delay module { eg.  AT4 (code 24) --- AT0 copied for User 4).  And then we wait for the next AT0 before we send the command to "RotateOut".  This way seems to guarantee that the target would enter the beam only once.

    In one commissioning version of polarControl, I asked Seth to specifically to let my DAQ system start many seconds before the target went in and wait many seconds after the target moved out.   Somehow, this way, the histogram (71 in the hbook file) would accumulate a lot of entries at the encoding position ~ 0.   This somehow screwed up the intelligence of my fitting routine fit.C which "smartly" tries to use the bin with the largest no. of entries as the candidate for the peak position of the Gaussian fit.  My simple solution is to make the lower edge of the histogram 71 to start from 10, rather than 0.

    Here is a Gpm plot which shows that the target moves to the "start" position and wait for a couple cycles (it's actually 5 s and then wait for the next beginning of the cycle) before it moves to the "end" position.

  27.  The problem with minimum momenta or Gγ values ( GCC = 0 )

    In the 2009 run,  I saw non-zero entries with minimum/lowest end of  momentum or Gγ spectrum in the histograms of the first page of the "expert" plots such as this.  After a while, I finally was smart enough to check and find that they all came from the 4 channels from a WFD and the GCC were always 0 for those 4 channels. Each WFD has one GCC input. Going to HITL2 and checking the GCC connection, I found the loose lemo (almost unplugged actually) near the GCC fan-out NIM module.  Plugging the lemo back has solved the problem.

    Of course, one has to make sure that the "RampBegin" parameter in the configuratio file "new_set.ini" is at about the place where the AGS ramp begins (~160 ns).  Because "RampBegin" is the time when the WFD starts to count GCC.  Setting "RampBegin" too late would prevent WFD from catching the earlier GCC signals and therefore the maximum momentum/Gγ would be smaller than what they should be.


  28.  User 4 → User 2 (late Feb. 2008):

  29.  The problem with the frequency multiplier:

  30.  Wierd GCC spectrum and polarization instability

    On Feb. 20, 2010, we started to see GCC plot such as asym_46212_p1.pdf, ie., it has a continuous spectrum at low GCC values instead of a unique bin at one high value.  This was accompanied with big χ2 (>>1) indicating large fluctuations in polarizations/asymmetries.  Cutting away the lower GCC values would get rid of the large fluctuations in asymmetries and χ2. At the end, this was found to be due to the loosened ECL cable that was/is plugged  to the LRS2367. After pushing it in all the way, the above problem has then disappeared.

     

  31.  Changing the CC32 cable could cause problem

    Apparently on Dec. 20, 2010, I unplugged and plugged back in the CC32 cables between the PC (cnipol2) and the CAMAC crate where LRS2367 was.  Then, the emittance plots were found to be emtpy or half Gaussian (or occassionally successful) and you'd see some cycles getting 10 times more events (like millions).  At the end, after I rebooted the PC, the problem went away.  Very likely, the CC32 driver (in Linux) is not plug-&-play and unplugging & plugging back the CC32 cable caused the weird problem.

  32.  

  33.  How to watch C-A TV (live) online ?
     

    [ Obsolete ! ] Click the "C-A Broadcast page"  → go to the top and click on the "small" scroll bar to go down to the bottom of that tiny page (very obscure !!) → click on "C-A TV"  → click the link at the top to watch the livefeed.ram (with realplayer etc.).
  34.  

  35.  Early or Late CBM or Booster_Input all are zeroes in the Krisch sheet ?
     

    Soultion: Restart "Krisch Server" (from https://startup.pbn.bnl.gov/servers ) for the right ppm-user (ie. when the proton beam is operated in AGS user 4, you probably don't want to do this when AGS is in user 1).
  36.  

  37.  Function Generator Issues

    Starting in 2014, we made use of the clock counts from John Morri's "Function Generator" in place of GCC during ramp measurements.  (The clock counts appear in channel 4 of the MUX_DAQ on the pet page.  GCC is in channel 3.)  

    [ Obsolete since 2021.  See the 2021 instructions instead !

    To start it, one may use "StartUp" to start the "Commissioning" version (Pol Timing) of the FunctionEditor
    → double-click on "AGS" → select "PolarizedProtons" → select "Timing" on the right → clock "OK" at the bottom
    → under "File", choose "Open"/"Library" → pick "
    150msTo650ms_50KHz" → click "Make Live" to send the function.

    To clean it (to save memory as there may not be enough cache memory for all 4 users.  When that happened, the function may not get sent next time when somebody reboots the cfe-htil2-pol chassis as insufficient cache memory doesn't contain that bit of information for the user that we care such as "user 4"),  do almost the same as above except that we pick
    "zeroFunc" to send. ]

    Diagnosis : 

    On the right hand side of the "FunctionEditor", click the "Pet Page" button and, put  your cursor on any of the "qfg.agspol-time" items.  Then, under "Device" at the top, click "Show Parameters".

    • Under "eventFormulaS", it should be "20" (as AGS T0 for all users).  In 2014, after a reboot by Al Marusic, as there wasn't enough cache memory, "20" wasn't restored for "eventFormulaS" and we got no clock count signals from this Function Generator. After one sets "eventFormulaS" to "20", one needs to click "Make Live" to make it take effect !

    • Another place to watch is "linkErrorM" which should be "None" when there is no problem.  "stateS", "stateM" and "statusM" should be "empty" and "statusM" is zero ("0x0").

    [ Obsolete since 2021.  See the 2021 instructions instead !

    New choice in 2017: 


    Instead of "150msTo650ms_50KHz", J. Morris provided a new function "50KHz1msecForRepeat_Start" which repeats the "1 ms pattern" 500 times (instead of one "500 ms pattern"). 

    • Usage of it is similar to "150msTo650ms_50KHz" but in addition, after loading "50KHz1msecForRepeat_Start" from the library, you need to click the "pet" and set "repeatCount" to be 500 (not 1 !). 
    ]

    In 2021: 

    Now, we go to the Pet page, select "AGS" → select "PolarizedProtons" → select "TriggersForRampMeas", at the top line that one may change settings,
    → put in "150" for the "start time (ms)" and put in "500" for "duration (ms)",

    after hitting return, you'll see (on the line below) :
    "start time (s)" as "0.15001,  "repeat count" changing to "500" and "last time (s)" changing to "0.65".   At 50 kHz, this will give us 25000 counts.


  38. Bunch hopping
     
    In the evening of May 31, 2015, I was called by MCR and we've finally realized that there was a problem of bunch hopping.  If I analyzed the events in the correct bunch (bunch 2 in my naming scheme), the banana plot seems fine.   But if I analyzed the events in the wrong bunch (bunch 22 in my nameing scheme), the plot contains background/prompt events.

    When I looked at the log file 67464.log, I saw:

    4 SP:+ 0 ( 9927) EV:   7167613 ( 919) 7799.36 TOT: 7.168*10^6
    4 SP:+ 1 ( 9929) EV:   6763603 ( 984) 6873.58 TOT: 13.931*10^6
    4 SP:+ 2 ( 9930) EV:     502382 ( 939) 535.02 TOT: 14.434*10^6
    4 SP:- 3 ( 9932) EV:   7172725 ( 949) 7558.19 TOT: 21.606*10^6
    4 SP:- 4 ( 9933) EV:     561891 ( 1049) 535.64 TOT: 22.168*10^6
    4 SP:+ 5 ( 9934) EV:    544142 ( 1032) 527.27 TOT: 22.712*10^6
    4 SP:- 6 ( 9936) EV:   6847385 ( 989) 6923.54 TOT: 29.560*10^6
    4 SP:- 7 ( 9938) EV:   7101107 ( 909) 7812.00 TOT: 36.661*10^6
    4 SP:- 8 ( 9939) EV:     550151 ( 1017) 540.95 TOT: 37.211*10^6
    4 SP:+ 9 ( 9940) EV:    534318 ( 972) 549.71 TOT: 37.745*10^6
    4 SP:- 10 ( 9942) EV: 7404133 ( 907) 8163.32 TOT: 45.149*10^6
    ...

    One'd see that when the bunch was in the wrong place, a lot of background/prompt events were collected which were ~10 times of the usual no. of events.  But when it was in the right bunch, only hundred of thousands of events were received.   When this problem happened, the intensities from cycle to cycle also fluctuated more than usual.
     
  39.  Emittance measurement shows "empty page" because the steps are always 0 ?

    On Dec. 14, 2016, when I started using polarimeter after almost 2 years, I found that the emittance measurement result was blank.  I checked the run.root in ~mcr/ado and found that there were many entries in histogram "h71" but all of them were "underflow", which (most likely) means that they were all "0".

    Peter Zimmerman and I went to HITL2 and we found that the MUX crate was turned on.  Peter checked and found that the -6V read ~0 though +6V read correctly.  So, we just turned off the power module in the NIM crate and then on.  And voila, when I tried the emittance measurement again, I saw the ~Gaussian curve !

    Afterwards, I've checked that when it works, all ±6, ±12 and ±24 volts for that power supply module should be on.

  40. Emittance blowup at flattop ?

    About 3 am on Feb. 3, 2017, MCR called me because they found that the 3rd picture on the left of this monitoring page (the grey area) showed decreasing no. of events sometime after 1.6 s (after AT0).  At the end, I've found that it's due to the emittance blowup after 1.6 s (after AT0) at flattop as shown in this IPM's emittance measurement plot (the left one at the bottom is the relevant one for the horizontal emittance when we were using vertical target 1).


  41. WinShift needs adjustment (noise vs χ2 stability)

    In the 2022 run, the polarization χ2 's (variation) were suddenly big (20, 30 or larger, instead of 1) --- which would make the polarization values unreliable due to its huge fluctuation.  The machine experts couldn't find anything wrong and changing pre-amp and targets didn't solve the problem (completely).  At the end, it was found that if I changed the WinShift (in the "new_set.ini" file) from 3 to 11 (setting considerably higher than 11 like 15 would see huge noise), the χ instability went away.

    But later, the RF signals jumped again and I needed to re-adjust WinShift to smaller values.


Back to my AGS CNI polarimetry page