The sky is really, really big. How do astronomers go from giant sky surveys to knowing what specific objects warrant further study? Are there automated processes to look for things with weird spectra or light curves or other stuff?

I remember volunteering online to look at stellar light curves for planetary transits, precisely because the task was hard to automate. I saw a couple of especially weird curves which I was able to flag for follow-up. Do y'all just sort data by hand?


ยท 3 ยท 11 ยท 7

@boisdevache @starkatt That's a very good question!

Many sky surveys are looking for a very specific thing. For example pulsar surveys scan the skies looking for pulsars. They (generally) look at the sky with radio telescopes and record data in a way that doesn't let you measure the absolute brightness of that part of the sky but do allow the detection of rapidly changing wide-band signals.


@boisdevache @starkatt

These pulsar surveys are simply not able to detect, for example, radio galaxies, even though they emit radio waves of the same frequency.

We then process the data, doing a highly computational search for periodic signals. Combing through these to get rid of all the junk terrestrial trap signals is pretty laborious but we have found several thousand pulsars this way.

We then, if we can (data volumes are huge) keep the data.


@boisdevache @starkatt

It's a very good thing that we keep the data, because our computational capacity is improving, and so is our ingenuity. The Parkes Multibeam Pulsar Survey found a large fraction of the pulsars currently known, and a fair number of those were found only upon reprocessing the data.


@boisdevache @starkatt
But to come back to your question, we know there might be things we are missing. A concrete example occurred with that pulsar survey.

On one of the reprocessing runs, Dunc Lorimer, Maura McLaughlin, and colleagues realized that for pulsars that spin slowly, we might have better luck looking for individual pulses rather than searching for periodicity. So they added that to the search pipeline.


@boisdevache @starkatt
In addition to some new pulsars, they found a few mysterious signals. One of them was a single burst that appeared to have come from a very distant galaxy, which meant it must have been stunningly bright for us to see it. Not a pulsar at all, this "Lorimer burst" stood as a solitary mystery for a few years. But having seen it, we knew to keep looking for bright single bursts.


@boisdevache @starkatt
In the last few years, we have started to find more such bursts, now called "Fast Radio Bursts": bright single flashes of radio that come from distant galaxies. They are an outstanding mystery and am active area of research. And they sat hidden in our survey data for years until a couple of brilliant astronomers thought of a way to look for them.

How many other mysteries are out there that we just haven't thought to look for?


This is so interesting, thank you for explaining it.

@boisdevache @starkatt if you'll forgive another long answer:

There are surveys done just as general-purpose sky surveys. Of course they reach have their limits, and they are done with at least one drievoudig goal in mind, but these are done with the idea in mind that they will answer questions nobody has yet thought of. The Sloan Digital Sky Survey is an extremely successful example of this.


@boisdevache @starkatt
But I personally have worked more with the Fermi gamma-ray telescope. It has a very wide field of view and operates in a mode where it scans the entire sky every few hours. The entire data set is public, and there is a catalog of sources.


@boisdevache @starkatt
But the way one does science with it is usually to come up with a question ("this source is weird, I wonder if it shows up in gamma rays? I wonder if it's changing?") and search through the data for the answer. One might build up a light curve, for example, that shows that the source unexpectedly brightened in gamma rays.


@boisdevache @starkatt
There are programs running to notice sources that do something surprising, but these generic programs and the catalogs are not as sensitive as a targeted search for the answer to a specific question.

This pattern, of a survey releasing all its data and of scientists going to that data set looking for answers to their questions, is a common way to get the most out of survey data. 4/?

@boisdevache @starkatt
It's not a new approach either: there are archives of photographic plates, now digitized, going back more than a century. If you are interested in a reasonably bright optical source, you can go back to these archives to look into its history.


@boisdevache @starkatt
In fact, this way of working, looking through sky surveys, hours back as far as the earliest use of plates by Cecilia Payne-Gapotschkin, Annie Jump Cannon, and others. It is well described in Dava Sobel's lovely "The Glass Universe", which describes how a group of women, hired because you didn't have to pay them as much as men, were key in turning astronomy into astrophysics.


@anne @boisdevache @starkatt
I read that book last year and loved it, can recommend.

@error_1202 @boisdevache @starkatt after reading the ebook I bought a paper copy to lend to colleagues; especially the women. Among many other things it's a good reminder that women are underrepresented for reasons that have nothing to do with our intelligence or skill.

@anne @starkatt question: what is a "drievoudig goal"?

@Felthry @starkatt that's swipe-to-type being helpful. I meant "specific".

It is also an example of an extremely annoying problem with the Android keyboard: I have installed multiple languages, so I can switch between them and get suggestions for whichever one I'm currently typing. But the keyboard happily suggests from all the languages, yielding random Dutch or French words even though I've told it I'm writing English.

@evilscientistca @boisdevache @starkatt I bet you have a different but equally interesting answer!

@starkatt In the not so distant past one hired women (because they were cheaper to hire <grumble) to look at each plate. When society finally figured out that doing things that way was wrong (the pay and treatment of the women that is) astronomy switched to grad students, who you could still underpay. 1/?

@starkatt The problem is (other than the ethics of the situation) that as technology has advanced, our ability to collect data has far surpassed our ability to analyze it in anything close to a timely fashion. 2/?

@starkatt For example, prior to the late 1980s and early 1990s, most astronomical imaging in the optical used film. Though better than the human eye, film is horribly inefficient at collecting light, the best films capturing about 10% of the light that hits them. 3/?

@starkatt modern sensors (CCDs) are much more efficient. Even the inexpensive ones are at least 60% efficient at collecting light and the high end professional ones are upwards of 97% efficient.

This efficiency is important because we see fainter and fainter things by increasing the integration time (for optical this equates to the exposure time). 4/?

@starkatt So with film if it took 1 hour to get to a certain dimness, modern CCD sensors can do the job in 6-10 minutes. This means that you can collect data 6 to 10 times faster than you could before.

This causes the problem that by hand it will take 6-10 times as long or needs 6-10 times as many people. 5/?

@starkatt Now as @anne said in her excellent reply surveys generally look for specific things and much of the data reduction and searching can be automated, depending on what you are looking for and trying to do. 6/?

@starkatt There are limits to automation however. It is relatively easy to teach a computer to find galaxies on a plate as they're extended objects not points like stars. It is much harder to use automation to classify those galaxies. So it's back to the grad students doing it by hand. However there is now such a huge amount of data available that there aren't enough grad students to do the work. 7/?

@starkatt Which is why astronomy is an early adopter of citizen science. It's relatively easy to train a lay person to classify a galaxy, for example. With the internet it's now also possible to have thousands of eyes on the project for little cost.

So computers are used to locate the galaxies on a plate, then the found galaxies are then the galaxies are sent to the trained public to classify them by hand. 8/?

@starkatt As an aside it is also important to remember that astronomers collect data for one specific thing that they are researching. There is a ton of information in that data that they're throwing away because it doesn't apply to their research.

This provides an opportunity for other astronomers to comb the same data for other information. 9/?

@starkatt as an example the Kepler space telescope was built and collected data with the goal of looking for planets around other stars. When I was in grad school a fellow student was using the same data to look for star spots.

Which is why I tell my students the most important thing as scientists we leave for future generations is not our discoveries, it's our data. 10/10

@evilscientistca thank you for this answer! I feel like I understand more fully now.

@starkatt @evilscientistca Same even. It's not really in my wheelhouse but I seriously appreciate that this stuff exists and like knowing how it works. ๐Ÿ’š

Sign in to participate in the conversation
The Vulpine Club

The Vulpine Club is a friendly and welcoming community of foxes and their associates, friends, and fans! =^^=