 | ctlearn | 20 Members |
| Deep Learning for IACT Event Reconstruction | 3 Servers |
| ctlearn | 20 Members |
| Deep Learning for IACT Event Reconstruction | 3 Servers |
| Sender | Message | Time |
|---|---|---|
| 12 Mar 2020 | ||
 @jonpsy:matrix.org left the room. | 10:08:38 | |
 shlydv | Ah cool | 10:09:29 |
 Tjark | Hi guys! Thanks for your interest In choosing CTLearn for GSoC 2020. First step would be to get familiar with the DL1-datahandler, especially with the writer.py. I will point you to some open dummy data from CTA and MAGIC later. | 10:21:05 |
| Tjark | https://github.com/cta-observatory/ctapipe-extra/blob/master/ctapipe_resources/gamma_test.simtel.gz | 10:23:51 |
| Tjark | There are also two MAGIC files in the ctapipe_resources folder. I think they actually contains real crab data. | 10:25:29 |
| shlydv | Thanks a lot for the resources! Took me quite some time to find the datasets on my own 🙂. | 10:58:04 |
| 13 Mar 2020 | ||
 @slal:matrix.org joined the room. | 03:59:40 | |
| 14 Mar 2020 | ||
 ayushvpaliwal joined the room. | 08:37:18 | |
| 18 Mar 2020 | ||
| Tjark | Hi @all! I hope the DL1 data handler tool didn't cause any troubles 😄. Did you guys managed to use this tool to produce a hdf5 file with the dummy data? With this file you should be able to launch CTLearn for some test runs. Further steps would be to read in trees from the MAGIC root file https://github.com/cta-observatory/ctapipe-extra/blob/master/ctapipe_resources/20131004_M1_05029747.003_Y_MagicCrab-W0.40%2B035.root using the uproot tool https://github.com/scikit-hep/uproot The tree that contains the pixel charges is ' Events.MCerPhotEvt.fPixels.fPixels.fPhot’. A great exercise would be to read in some random events and convert the 1D array (hexagonal grid) to a 2d square images using the image mapper in dl1 data handler. You need to select the ‘MAGICCam’ as the camera type. In case you need more literature: https://arxiv.org/abs/2001.03602 https://arxiv.org/abs/1912.09898 https://arxiv.org/abs/1912.09877 https://arxiv.org/abs/1709.05889 https://www.cta-observatory.org https://veritas.sao.arizona.edu/ https://magic.mpp.mpg.de/ https://www.mpi-hd.mpg.de/hfm/HESS/ | 21:00:36 |
| Tjark | (edited) ... in trees ... => ... in a couple of trees ... | 21:02:50 |
| 21 Mar 2020 | ||
 ganeshtero joined the room. | 12:31:07 | |
| 23 Mar 2020 | ||
| ayushvpaliwal | Yes, sir, I got quite familiar with dl1DH. And was able to produce a HDF5 file from dummy data. The given exercise was interesting. I made the required conversion. I have further doubt on same. Sir, there we are using five different methods of conversions. Can we train some data after applying these processing on any neural network model to determine which is best of them all? Or their use is event dependent? | 18:02:36 |
| 24 Mar 2020 | ||
| Tjark | Awesome! What is stored in the hdf5 are the calibrated charges of the PMTs in the camera. In most of the IACT cameras this layout is a hexagonal grid. To plug our data to CNNs we need to interpolation our events to square the images (see https://arxiv.org/abs/1912.09898). The image_mapper is performing this task. For now, you can just use the bilinear interpolation as the mapping method. With this dummy data you should be able to start training the CTLearn models, but the results are meaningless, because it's just a handful gamma events. However, it's a good exercise to get familiar with CTLearn. A more important exercise would be to read in some MAGIC events from the root file using the uproot tool and then plot the squared images of these events. Feel free to share results (event images) at earlier stage 🙂 | 10:32:19 |
| Tjark | (edited) ... start training the ... => ... start train the ... | 10:32:54 |
| shlydv | Redacted or Malformed Event | 10:50:15 |
| 25 Mar 2020 | ||
| shlydv | Hey Tjark sorry for the late reply. I was able to get the hdf5 output file. I'm getting familiar with the image mapper and uproot. Will share squared images soon! | 14:00:32 |
 daria.ge joined the room. | 14:55:46 | |
| 26 Mar 2020 | ||
| ayushvpaliwal | Dear Tjark, I have a doubt. Till now, I got familiar with DL1DH. And reading the stuff regarding uproot package. I understood what we can do with both packages, but I want to know what step by step process we will follow to make ROOT enable. Can you please explain rough map for this ?? | 17:35:30 |
| Tjark | Great @shlydv! | 18:28:35 |
| Tjark | Since most of you got familiar with the DL1 data handler, we can dig deeper and in a more technical way into the project. A step by step would be: 1. Implement a DL1DataWriter (or modifiy the current one) for the root files of MAGIC and VERITAS. For MAGIC you "only" need to change the event source with the MAGIC event source, which is already in place (https://github.com/cta-observatory/ctapipe_io_magic). For the VERITAS file you would need to open the file with the uproot packages to retrieve the events "by hand". 2. Produce hdf5 files for MAGIC and VERITAS, which can be read directly from the DL1Reader (without any changes). 3. Test the hdf5 files with the existing models in CTLearn. Implement new models in CTLearn, which are especially design for the array layout of MAGIC (2 IACTs) and VERITAS (4 IACTs) 4. (If time allows) Compare to existing analysis chains like Random Forest or Boosted Decision Trees. 5. Update code, documentation and plots in Github Hope that helps. Don't hesitate to ask further questions in case something is unclear! | 18:57:05 |
| shlydv | Thanks for the info! Will start working on it. Also, do we have to submit a PR before the application deadline i.e. 30 March or just the proposal? | 19:13:22 |
| ganeshtero | proposal deadline is 31st March right? | 19:14:07 |
| ayushvpaliwal | What we will do for FACT, HESS, CTA? | 19:31:59 |
| ayushvpaliwal | Our first step was to run and verify existing DL model and for preprocessing we are using DL1DH. Right? Second was to make it ROOT enable using uproot. How we will do this ? I stuck here. And at last we will make it multi telescopes enable. Right? Please guide me on this | 19:43:38 |
| Tjark | The proposal deadline is next Tuesday (31.03). Yes you should submit a PR (including your progress over the last weeks; for example the squared images for MAGIC) and a proposal before Tuesday. | 19:56:33 |
| Tjark | CTA is already in place! For FACT and HESS we don't have access to data. | 19:59:08 |
| Tjark | You need to implement a class similar to the DL1DataWriter class in writer.py. Instead of reading the data from a simtel file, you need to read the data from the root file. For MAGIC you only need to change the event source (from the default CTA to the MAGIC one), since a plugin is already available. | 20:07:31 |
| Tjark | I think I'm not getting your question. What do you mean with multi telescopes? | 20:08:31 |
| ayushvpaliwal | What mentioned in idea page was initially we will make it to enable data in ROOT format from specific IACT. And afterwords from given collection of IACTs | 20:12:00 |
| Tjark | Okay, I see! MAGIC is a collection of two IACTs and VERITAS is a collection of four IACTs. However, the data structure is the same for the files of each IACT in their array (collection). Shouldn’t cause any troubles scaling up for „multi telescopes“. | 20:41:46 |