Entering ZTF-O4
ZTF was seen as a pre-cursor facility for LSST (now the Vera Rubin Observatory) in time-domain astronomy. It was funded in about equal parts by NSF ($11M) and the partnership. First light was achieved in late Fall of 2017 and the survey began in Spring of 2018. ZTF ran through Sepember 30, 2020. It was the first survey to put out real time alerts. This facility combined with professional and frequent data releases by IPAC nurtured and nourished brokers around the world. ZTF was wildly successful and separately LSST was delayed. ZTF Phase II (ZTF-II) was funded again in the same manner as ZTF. NSF contributed $5M and the partnership contributed about the same amount of funds. ZTF-II came to an end on September 30 of this year.
As you can see from the summaries and statistics below, ZTF- now a mature project - continued to be a great success across the board: in stellar astronomy, in supernovae & TDE and transient science. For the first two, the focus shifted from small samples to very large samples and for the third the focus was on detection and pursuit of fast transients. The alert stream from ZTF is now regarded as a reliable fixture in the time domain astronomy landscape!
A new partnership and supplemental funds from NSF ($1M) will keep ZTF going through LVK run O4. The project team is exploring various options to ensure ZTF operations for CY 2025-2028 during which period Rubin would be in routine operation. We see a great synergy between LSST (slow but very sensitive) and ZTF (shallow but high cadence).
Shri Kulkarni
ZTF Principal Investigator, Professor of Astronomy and Planetary Sciences, Caltech
Matthew Graham
ZTF Project Scientist, Research Professor of Astronomy, Caltech
At a Glance
A quick snapshot of ZTF in numbers
Institutional partners
Researchers & students
Partnership publications
Community publications
Nationalities
Summer school participants
Operational nights
Science observations
Science images
Engineering Maintenance
Operational Overhead
Filter exchanges
NEA & comets
Tidal disruption events
Ultra-compact binaries
Note: The time span used to generate the statistics above is Oct 2018 to Sept 2023.
Science & discoveries during ZTF II
The leads of the science working groups in ZTF share the most significant moments for their teams during ZTF II
fPhysics of Supernovae and Relativistic Explosions
The Bright Transient Survey
crew continues to collect supernovae, and it now has over 8000
classified supernovae. This fantastic influx of transients is feeding a wide variety of science
and in the past three years we have published papers both on individual nearby SNe like
SN2020jfo
and more recently the upcoming paper by
Erez et al. on
SN 2023ixf.
The group has also compiled some more systematic samples, like the
SNe Ia-csm by Yashvi Sharma
and the very comprehensive sample papers
by the Superluminous supernova sub-group.
A few papers made it to high-profile journals like Nature, for example the discovery of a
radio Type Ia supernova, the discovery of a jetted
TDE as well as Avishay et al. paper
which together with this paper led by Daniel Perley established an entirely new class of Type Icn supernovae.
Anna Ho has been leading the emerging field of fast transients
and Kate Mcguire's group in Dublin and also Adam Miller's group in Chicago took care of our peculiar SNe Ia, and published a
series of papers on the topic.
Some members of the team wrote up their samples and PhD theses and moved on to other teams, congratulations for example to
Rachel Bruch at Weizmann Institute of Science, Israel.
Galactic & M31 science
One productive aspect of ZTF-II was the emergence of an active group finding and conducting follow-up observations of interesting cataclysmic variables (CVs). This group involved international and domestic collaborations between Caltech (Tony Rodriguez, Zach Vanderbosch, Jan van Roestel [now in Netherlands], Ben Roulston [now in New York] and Andrew Drake), U. of Washington (Paula Szkody, Eric Bellm, and Keaton Bell [now in New York]), UCLA (Michael Rich), Boston University (JJ Hermes), and U. of Warwick (Boris Gaensicke and Keith Inight). The research was aided by the use of the Fritz Marshal and the ongoing activities of the machine learning group, involving collaborations with Ashish Mahabel (Caltech), and Michael Coughlin and Brian Healy (U. of Minnesota). Among the results are the discovery of several thousand new CV candidates, including AM CVn stars, and those with magnetic white dwarfs. The use of the ZTF light curves to find orbital periods along with X-ray to optical survey data and follow-up spectroscopic and photometric data from Keck, Palomar, Apache Point, and Lick Observatories, have resulted in an array of papers providing needed information on the numbers of these systems in our Galaxy and the evolutionary paths that brought them to discovery. An upcoming catalog will allow continued future follow-up.
In addition, smaller collaborative groups have continued to expand upon the discoveries and characterizations of several different stellar populations. These populations include massive and rapidly rotating white dwarfs thought to be the products of white dwarf mergers (Ilaria Caiazzo), ultra-compact and evolved binary systems (Kevin Burdge, Tom Prince, Kareem El-Badry), white dwarfs hosting tidally-disrupted and transiting planetary debris disks (Zach Vanderbosch, JJ Hermes, Soumyadeep Bhattacharjee, Boris Gaensicke), pulsating and other variable stars within globular clusters (Chow-Choong Ngeow), the central stars of planetary nebulae (Albert Kong), young stellar objects (Lynne Hillenbrand, Michael Kuhn, Avery Wold), and M-dwarfs and dwarf carbon stars (Rocio Kiman, Ben Roulston). Several papers and catalogs are forthcoming from these efforts.
TDEs & AGNs
As with most science, the real energy and progress in the our team has been driven by early career scientists. Code development, scanning, triggering follow-up, observing and building samples has always been a collaborative effort in our group, and for ZTF-II we would particularly like to highlight the hard work of grad students Erica Hammerstein (UMD), Jean Somalwar (Caltech) and Yuhan Yao (Caltech, now Berkeley). Thanks to their efforts, we have ~50 new TDEs in ZTF-II, representing ~half of all known optical TDEs.
On the back of this large sample, there were a number of important papers to come out of our group during ZTF II. Erica Hammerstein led a comprehensive analysis of our ZTF-I TDEs, and identified the new spectroscopic class of “featureless TDEs”. Yuhan Yao led a combined analysis of ZTF-I and ZTF-II TDEs, producing the first large sample estimates of the TDE luminosity function and volumetric rates. There was also close collaboration with the MMA working group, including the detection of three different possible neutrino-TDE associations led by Robert Stein for AT2019dsg, Simeon Reusch (DESY) for AT2019fdr, and Sjoert van Velzen (Leiden) for AT2019aalc. Igor Andreoni (UMD), as part of his search for rapid transients, identified the first optically-discovered jetted TDE. And Matthew Graham (Caltech) identified candidate optical counterparts to binary black hole mergers detected by LIGO. There is still much to learn about TDEs, and plenty of work is still ongoing. Most recently, there have been a couple of ZTF TDEs that exhibited repeated flares. A paper about it should shortly be on arxiv! We don'tdoubt that ZTF TDEs will continue to surprise us in the future.
Solar System Science
The Solar System Working Group is a small team, but we can still look back and find several accomplishments during ZTF II. Our group published papers detailing the use of state-of-the-art software techniques for solar system science, including a new method tocharacterize the rotational light curves of asteroids, and the use of deep learning for comet discovery. Our understanding of the Solar System benefits from the study of extreme objects, and our team reported on kilometer-sized asteroids that appear to have "super-fast" periods, comet P/2020 HS, one of the weakest comets in the inner-solar system, and asteroid Ayló'chaxnim, which is the first asteroid known with an orbit that is fully interior to the orbit of Venus. With ZTF II, we discovered 107 near-Earth objects, 23 cometary outbursts, and also found comet C/2022 E3, the photogenic comet that seemingly broke the Internet as the "Green Comet" not seen for tens of thousands of years. That's a comet we'll never forget!
Multi-messenger Astrophysics
ZTF-II was an exciting time for the multi-messenger astronomy working group, across our three flagship programs. The advent of Fritz, with its host of novel MMA features, has been a particular game-changer for our team. The ZTF neutrino program continued apace, and led to the identification of two separate TDEs as probable neutrino sources (see https://www.nature.com/articles/s41550-020-01295-8, https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.128.221101). The quest to find short gamma-ray burst (GRB) afterglows led to the unexpected discovery of a supernova counterpart, confirming that some short GRB have a collapsar origin. More recently, we found a supernova associated with the nearby, bright long GRB 230812B (Srinivasaragavan et al. 2023, in prep.). Our search for kilonova during O3 led to constraints on the kilonova luminosity function and properties of two neutron star–black hole mergers. We also developed a novel pipeline for serendipitous discovery of kilonova-like rapid transients, ZTFReST. ZTFRest has found many rare transients, including GRB and orphan afterglows, and even a jetted TDE. Our successes have been supported by the highly collaborative nature of our working group, with active participation from teams in Germany, India, Sweden, the UK, and across the US. This diverse group, with varied experiences, backgrounds and interests, have enabled new discoveries, and impactful papers. While ZTF-II may be ending, we are confident these relationships will endure and continue to lead to fruitful future cooperation!
Machine Learning
Machine learning (ML) has enabled powerful inference for astronomical data at scales and speeds surpassing human limitations. During ZTF II, we achieved significant advancements in the development of machine learning algorithms to study the time variable sky. We have successfully developed a workflow for training machine learning algorithms to make classification predictions for ZTF sources. Furthermore, we have built a deep learning model to automatically identify BTS sources, saving scanners’ time and expediting follow-up. We have harnessed the state-of-the-art transformer architecture to yield 95% accuracy in quasar vs galaxy vs star selection based on ZTF variability and photometric colors. The in-class anomaly detection pipeline is complete and is being run on SCoPe+ fields, with the best candidates selected for spectroscopy. To support multimessenger science, we have completed the development of the first autonomous planning agent for following up kilonovae, with an accuracy close to human astronomers. We have also created the most accurate observing scenarios to date, aiding in the preparation of multi-messenger searches with ZTF. The Quarter Century Sky project has streamlined its GUI with Django-React, optimized data storage and querying using PostgreSQL and Redis, implemented security measures, and enhanced functionality for bulk data uploads and querying. Finally, our engagement with the wider community includes the launch of ZARTH, the Pokemon GO like android game for transients, which has been played by people from close to 50 countries.
In the course of ZTF II
Behind the science are the people and their personal stories...
My daughter Violet was born in June 2022 and has quickly become the center of my life. It’s a challenge being a dad at this age, and trying to keep on schedule at work while handling a now-toddler who has to run around and touch everything, but it’s been worth it. It’s a good thing I’m an astronomer though, otherwise, all the late nights and lack of sleep over the past year would have been much harder.
-- Reed Riddle (ZTF Robotic Lead, Caltech, USA)
In the course of ZTF II, I attended my PhD graduation ceremony at Caltech and moved from California to Ithaca NY to start a faculty job at Cornell University. I also got a cat.”
-- Anna Ho (Assistant Prof, Cornell University, USA)
In the course of ZTF, every morning for the past five years I started the day by discovering a couple of new supernovae. There is no better way to start a day...
-- Jesper Sollerman (Professor, OKC, University of Stockholm, Sweden)
In the course of ZTF II, my third daughter, Matilde, was born in Oct 2021. In June, I moved back to Italy after a ~10-year research journey with stops in Munich, Belfast, and Stockholm. I started a faculty position at the University of Ferrara in September last year.”
-- Mattia Bulla (Assistant Professor, University of Ferrara, Italy)
In the course of ZTF II, I became an assistant professor at Northwestern University and the Director of the LSSTC Data Science Fellowship Program.
-- Adam Miller (Assistant Professor, Northwestern University, USA)
In the course of ZTF II I moved to Dublin for my PhD, getting to know many wonderful people both in the working group and ZTF as a whole. I also moved to La Palma to learn more about the observational side of astronomy by getting to work and observe with the Isaac Newton Telescope.
-- Jacco Terwel (PhD student, Trinity College Dublin, Ireland)
In the course of ZTF II I got married:)
-- Shreya Anand (PhD student, Caltech, USA)
In the course of ZTF II, I defended my PhD thesis in July 2022! It was titled 'A portrait of the binary compact merger as a young: Short GRB, Gravitational wave, Afterglow, and Kilonova'. I also started a position at Caltech as a postdoc.
-- Tomas Ahumada (Postdoc, Caltech, USA)
In the course of ZTF II I had two major achievements in my life: one personal and one professional. Professionally, I fulfilled my long-sought goal of obtaining the ERC Starting Grant to study common envelope transients. This has allowed me to move to the University of Barcelona with a tenure track position. I am starting my own research team in observational time-domain astronomy, and bringing this new field of expertise to the university. Personally, I also fulfilled my dream of having a family with two kids. My second son Leo was born healthy and happy (or as happy as a newborn can be). After these new beginnings, new challenges lie ahead to nurture and grow these precious seeds to allow them to flourish.
-- Nadia Blagorodnova (Asst Professor, Universty of Barcelona, Spain)
Outside the spotlight....
Authors of leading discoveries that grabed the media attention, speak about key team members who remained behind the spotlight...
ZTF makes discovery of a rare cosmic lunch
The person behind the scenes that I'd like to mention as instrumental to the discovery of AT2022cmc is Roger Smith, the detector and electronics lead at Caltech Optical Observatories. Roger and his team designed and built the ZTF camera. Importantly, they took care of it numerous times when it needed attention during the course of the survey. In January 2022, small crystalline particles called "zeolites" were found that were contaminating the camera. Their presence presented a potentially fatal risk to the electronics. Roger and other engineers brought ZTF back to Caltech campus, dismounted the camera to remove all the zeolite particles, ran additional tests, and mounted it back on the telescope at Palomar in record time. Their working schedule was relentless because in astronomy every single night matters, as this story demonstrated. On the night after ZTF started observing again, the one-of-a-kind fast transient called AT2022cmc was detected by ZTF. AT2022cmc is the first relativistic tidal disruption event identified by an optical survey, in which a luminous beam is launched when a star was pulled apart by a massive black hole's gravity, and is also the furthest of such events discovered to date. If ZTF started observing again even a couple of days later, we would have likely missed this exceptional cosmic drama. Thank you, Roger!
Radio signals reveal origin of a supernova
In order to make sense of what ultimately would be the first radio-detected thermonuclear supernova, I had to call in a lot of help. So there are many people without whose help this study would not have been as interesting as it turned out to be. But I think it’s fair to say it may not even have existed in any shape or form at all without second author Joel Johansson. The project was shelved for quite some time because we didn’t really know what we were looking at, and it took Joel’s conviction that it was in fact a thermonuclear supernova (of a kind not observed before) to get the ball rolling properly. To me, this study has been a great example of how powerful it can be to seek out and work with people with different scientific backgrounds and expertise, rather than go at it alone.
Two faced star exposed
I would like to give a shout-out to the work of Daniel Perley in developing the reduction pipeline “LPipe" for the LRIS spectrograph. Having a reliable pipeline for reducing spectroscopic data is crucial and Dan’s pipeline has been an invaluable asset for the ZTF group at Caltech, as it runs fast and smoothly, enabling an almost real-time reduction during the night. LRIS spectroscopy was the decisive dataset in the discovery of the double-faced white dwarf, as it revealed that one face of the white dwarf is covered in hydrogen, while the opposite face is covered in helium. When I discovered this strange behavior, I needed more spectra to confirm that the modulation was stable and to obtain phase-resolved data with a higher signal-to-noise ratio. In the meantime, however, a new detector was installed in the red arm of LRIS, and therefore the pipeline needed to be updated. I reached out to Dan and he swiftly modified the pipeline, so I was able to analyze the spectra within a couple of weeks. The work of developing a reduction pipeline is not always properly recognized, but reliable and fast pipelines are extremely important for maximizing the scientific return of an instrument, and Dan’s is one of the best I have had the fortune to use.
NASA’s Fermi Spots a Supernova’s ‘Fizzled’ Gamma-Ray Burst
Gamma-ray bursts (GRBs) are detected almost every day, but it is extremely difficult to pinpoint their optical counterpart. By August 2020, we have used ZTF several times in the hopes of finding the elusive afterglow. At 11 pm PT time, Igor Andreoni realized a short GRB was detected by Fermi, and he interrupted ZTF schedule and triggered observations on the GRB error region. Multiple people around the globe started scanning candidates in real-time and Ana Sagues-Carracedo led our first GCN, with help from folks in Germany and India. Once we secured the afterglow, Shreya Anand coordinated key observations with collaborators, while managing the resources we already had available like LCO and P200.
This burst was controversial as it showed long GRB properties while lasting only a 0.65 sec in the rest-frame. Leo Singer was instrumental in leading a Gemini Director's Discretionary Time proposal to elucidate the nature of the burst. I remember how we stayed up late in the midst of the pandemic, coding and writing over the phone, checking references, and making plots. It was one of the highlights of my PhD, incredibly insightful and exciting at the same time. Leo guided me through the process, sometimes calling Mansi or Brad to check on SN parameters or configurations of the telescope, and later introducing me to new analysis techniques and even better, bugging Michael Coughlin and theorists like Geoff Ryan to help me get the numbers right.
After long and late hours, coordination with 50+ authors and data from 10+ facilities, we submitted our paper to Nature Astronomy along with Bin-Bin Zhang's work on the same object. But that's *not* the end of the story! Leo and Brad were able to excite NASA about our discovery: so much so that they decided to make a video of the GRB and allowed me to narrate the Spanish version of the video. That felt surreal. As surreal as a 0.65 sec duration collapsar.
Astronomers discover extremely warped supernova
We were thrilled to find the first strongly lensed Type Ia supernova with ZTF last year, SN Zwicky. The lens, a galaxy in the line of sight, split the supernova light into four images separated by just a fraction of an arcsecond and were ultimately spatially resolved with laser-guided adaptive optics at Keck in the Near-IR and HST at optical wavelengths. One key measurement of SN Zwicky was carried out by OKC student Ana Sagués Carracedo: the difference in arrival times between the images, known as “time-delays”. Before SN Zwicky, the prevalent impression in the community was that an accurate measurement of the time-delays between the supernova images would not be possible unless we could carry out follow-up observations of the resolved images. In other words, it requires high spatial resolution over multiple epochs from repeated observations with Keck or HST. These are very expensive and hard to get, especially for a large sample. Instead, Ana showed that a single high-resolution image from Keck of SN Zwicky, identifying the four images and their relative fluxes, combined with the unresolved ZTF images was sufficient to measure time-delays in this case. Ana’s results are very encouraging and show the potential to uncover important cosmological information from gravitationally lensed supernova based on ZTF data, or similar.
ZTF is supported by the National Science Foundation and a collaboration including Caltech, IPAC, the
Weizmann Institute for Science, the Oskar Klein Center at Stockholm University, the University of
Maryland, Deutsches Elektronen-Synchrotron and Humboldt University, Lawrence Livermore National
Laboratory, the TANGO Consortium of Taiwan, the University of Wisconsin at Milwaukee, Trinity College
Dublin, IN2P3, University of Warwick, Ruhr University Bochum, Northwestern University and Cornell University . Operations are conducted
by COO, IPAC and University of Washington.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the
author(s) and do not necessarily reflect the views of the National Science Foundation.