Taking that picture of a black hole required massive amounts of data

[Cal Jeffrey]

In context: Eight radio telescopes trained at the center of the Messier 87 galaxy each recorded 350TB of data per day for one week. That provided the scientists with five petabytes of data stored on high-performance helium-filled hard drives to process. An amazing cache by anyone's standards.

On Wednesday, we reported that scientists had taken the first image ever of a black hole. It was big news in the astronomy community. It was also apparently big news in the data-hoarding community as well.

Subreddit DataHoarder was buzzing after one of the scientists working on the black hole project tweeted a picture of his colleague, Katie Bouman, posing with some of the hard drives that held the image data.

She stood behind eight towers that appeared to hold eight drives each. Presumably, this was just one cache as 64 HDDs could only hold 5PB if they were 80TB capacity, something drive makers have not achieved yet.

“The Black Hole photo is very impressive, but I'm more interested in where they got these 80TB drives,” said one Redditor.

This is her with the hard drives containing the image data for the Black Hole. pic.twitter.com/unOud3msk2

— Muhammad Karim (@mkarim) April 10, 2019

Another poster posited that they used 12TB drives and that this was only one-eighth of the data.

"Well that makes sense as the data was captured across 8 different telescopes, so this is 1/8th of the whole setup," said ThomasTheSpider.

The black hole image was put together using data from eight radio telescopes from around the world. Each telescope gathered massive amounts of information on its own. All totaled the scientists were dealing with more than five petabytes of data — enough to store 5,000 years worth of MP3s.

Or as one data hoarder put it, “Black holes are cool, I guess, but imagine all the [lossless Blu-Ray files] you could store on those bad boys, in RAID 1 no less.”

Regardless of how many drives were used, there is a logistical problem with sharing this much information. Obviously, it can’t be sent via the internet. Instead, the drives were crated and flown to Boston and Germany for processing.

The data was then pieced together by an algorithm written by Katie Bouman. Blank spots were filled in by the software. Bouman, who is an assistant professor at the California Institute of Technology explained the process back in 2016 when she was still an MIT student (above).

“If we want to see the black hole, we need a telescope the size of the Earth [like a disco ball],” Bouman said. In other words, mirrors covering the entire planet, or at least a hemisphere.

However, having only eight mirrors limits what can be seen. So data was collected from each telescope every day throughout the day for an entire week.

“As the Earth rotates, we get to see other new measurements,” she continued. “In other words, as the disco ball spins, those mirrors change locations, and we get to observe different parts of the image. The imaging algorithms we developed fill in the missing gaps of the disco ball in order to reconstruct the underlying black hole image.”

This is why the image took up five petabytes of data. Could you imagine the data involved if it were possible to cover the Earth with mirrors to "see" the whole image?

Permalink to story.

https://www.techspot.com/news/79637-taking-picture-black-hole-required-massive-amounts-data.html

 

[FF222]

What's worse, that as explained by her in her video, this was a guided process that preferred outcomes which matched a pre-set concept of how the image should look. Which in turn means, that the black hole might not look at all like in this image, and the image was only generated this way from all the noisy data because of the built in bias in the process. So, when we'll be able to take better images of for ex. a black hole that's closer to us, it might turn out that black holes don't look at all like in this picture.

 

[TheDevopsGuy]

This just fascinates me how technology has become so advanced quickly, remembering my childhood 20 gigabytes were considered a lot and look at the stage were at now. Time sure is flying by..

 

[pcnthuziast]

Let the msm tell it and apparently Katie Bouman did it all by herself.

 

[seeprime]

Let the msm tell it and apparently Katie Bouman did it all by herself.

Agreed. A team leader is not the entire team,

 

[Jeff Re]

I want to know if they backed up the data before shipping it.

 

[Squid Surprise]

What's worse, that as explained by her in her video, this was a guided process that preferred outcomes which matched a pre-set concept of how the image should look. Which in turn means, that the black hole might not look at all like in this image, and the image was only generated this way from all the noisy data because of the built in bias in the process. So, when we'll be able to take better images of for ex. a black hole that's closer to us, it might turn out that black holes don't look at all like in this picture.

It's not quite as abstract as you are leading us to believe... the "pre-set" concept of what a Black Hole should look like is based on tons of previously gained data... this isn't just a bunch of grade-schoolers shooting in the dark.

It's a far more complicated process - and it's almost certainly a very accurate image. Will be nice to see the one they take of closer holes though...

 

[Spudnation]

Am I the only one who thinks the image is a little underwhelming...?

 

[Vulcanproject]

I want to know if they backed up the data before shipping it.

They didn't. The amount was so large and the cost of duplicate drives beyond the budget. What they got they flew around the world and hope it got there safely.

The BBC did a fantastic hour long show on this entire endeavor and it aired the day the picture was revealed. It was pretty fantastic, a great deal of the hardware are was shown off.

'How to see a black hole: The universe's greatest mystery.' Available on BBC iPlayer. Try a VPN or some such if you outside the UK.

 

[Lew Zealand]

What's worse, that as explained by her in her video, this was a guided process that preferred outcomes which matched a pre-set concept of how the image should look. Which in turn means, that the black hole might not look at all like in this image, and the image was only generated this way from all the noisy data because of the built in bias in the process. So, when we'll be able to take better images of for ex. a black hole that's closer to us, it might turn out that black holes don't look at all like in this picture.

It's not quite as abstract as you are leading us to believe... the "pre-set" concept of what a Black Hole should look like is based on tons of previously gained data... this isn't just a bunch of grade-schoolers shooting in the dark.

It's a far more complicated process - and it's almost certainly a very accurate image. Will be nice to see the one they take of closer holes though...

No no no.

The armchair scientists here at TS totally know better than the scientists who do the actual work. It's, like, way more accurate to come up with a superficial opinion in 30 seconds than spend a decade or more of your life generating, planning, and executing experiments.

 

[Lew Zealand]

Am I the only one who thinks the image is a little underwhelming...?

Yeah, that was my first reaction. Here it is with a scale, which really doesn't change that impression but at least it's more than just a blob:

https://xkcd.com/2135/

 

[VitalyT]

I am aroused, that black hole does it for me. And I'm beginning to understand what managed to move Einstein in his elderly years.

 

[FoxxiFurr]

But they didn't take a picture of the black hole in our galaxy, they took a picture of the one in M87. This article isn't credible in any sense of the word.

 

[Lew Zealand]

But they didn't take a picture of the black hole in our galaxy, they took a picture of the one in M87. This article isn't credible in any sense of the word.

Eh? Reading comprehension?
What does the black hole in our galaxy have to do with this article?

 

[sdms96825]

All that data for this shi**y photo? What a waste of $$$$.

 

[LeroN]

It looks like some new kind of a scientist's fake. The presentation seems done in a kidden garden.

 

[DukeJukem]

Why are we wasting money studying something that will simply kill us if it/we get close enough. it's like an ant studying a wind tunnel thats on with millions of spinning blades inside. entirely pointless and a waste of time.....wtf. cure something.

 

[Squid Surprise]

Why are we wasting money studying something that will simply kill us if it/we get close enough. it's like an ant studying a wind tunnel thats on with millions of spinning blades inside. entirely pointless and a waste of time.....wtf. cure something.

Well, here's the thing.... one can never predict what advances scientific research will lead to... for instance, the invention of Velcro came from NASA scientists trying to find an adhesive material that would work in zero gravity...

We fund all sorts of science that seems "useless" just because of things like this... for all we know, a really cool invention might spring from this...

And it wasn't YOUR time being wasted - other than the few seconds you "wasted" reading this article... I say "seconds" because I'm fairly certain you didn't read the whole thing...

 

[wiyosaya]

This just fascinates me how technology has become so advanced quickly, remembering my childhood 20 gigabytes were considered a lot and look at the stage were at now. Time sure is flying by..

In my early computer days, 20-megabytes was considered a lot.

 

[wiyosaya]

Am I the only one who thinks the image is a little underwhelming...?

All that data for this shi**y photo? What a waste of $$$$.

It looks like some new kind of a scientist's fake. The presentation seems done in a kidden garden.

Why are we wasting money studying something that will simply kill us if it/we get close enough. it's like an ant studying a wind tunnel thats on with millions of spinning blades inside. entirely pointless and a waste of time.....wtf. cure something.

Whatever each of you used to post these messages was made possible by science. Often, science leads to the unexpected that the layman does not yet see...

 

[J spot]

Am I the only one who thinks the image is a little underwhelming...?

All that data for this shi**y photo? What a waste of $$$$.

It looks like some new kind of a scientist's fake. The presentation seems done in a kidden garden.

Why are we wasting money studying something that will simply kill us if it/we get close enough. it's like an ant studying a wind tunnel thats on with millions of spinning blades inside. entirely pointless and a waste of time.....wtf. cure something.

Whatever each of you used to post these messages was made possible by science. Often, science leads to the unexpected that the layman does not yet see...

Those were some of the most ignorant comments, surprised that such people would be on a tech website. Extreme lack of curiosity or understanding of why others would like it. And even the typical, "Why can't these astronomers cure something?" Basically people who think that these things are done fore likes on social media.

Just the way they were able to get such a close up shot by essentially creating a huge radio telescope using the rotation of the Earth to fill in the different spots was really amazing.

The reason it looks blurry is because it really is just a zoomed in dot inside of what would be a regular size pictere. That's how far away it is.

 

[Wahidsatu]

Well, I guess Nokia is heading in the right direction for once.

 

[FF222]

It's not quite as abstract as you are leading us to believe..

Me leading you to believe? But it wasn't, but she explaining the bias process. I just made a logical conclusion based on what she said. So, IF what she says is correct, then what I'm saying is ALSO correct.

. the "pre-set" concept of what a Black Hole should look like is based on tons of previously gained data...

Which if used as a bias, will ensure that we can only get results that mostly confirm the already existing data - even if reality is different. That's the actual point I made.

It's a far more complicated process - and it's almost certainly a very accurate image.

That smear blob? I mean, if it would be a really crisp and detailed image, then chances would be lower that all of it was only a result of the bias. But this smear is not definitive by any means. With all the bias added to the filtering process, you could generate this smear even from totally random noise. Which is essentially what they supposedly did - except the data wasn't totally, just partially random noise, which however overwhelmed the actual data. That's the reason why the biased filtering was needed.

Now, IF the bias was correct, the image still could be "correct". But if the bias was incorrect this is most likely a totally false image. Or at least contain the same possible inaccuracies as the original data used to bias the filtering algorithm.

 

[Reachable]

Am I the only one who thinks the image is a little underwhelming...?

All that data for this shi**y photo? What a waste of $$$$.

It looks like some new kind of a scientist's fake. The presentation seems done in a kidden garden.

Why are we wasting money studying something that will simply kill us if it/we get close enough. it's like an ant studying a wind tunnel thats on with millions of spinning blades inside. entirely pointless and a waste of time.....wtf. cure something.

Whatever each of you used to post these messages was made possible by science. Often, science leads to the unexpected that the layman does not yet see...

... and there's a human need to explore. I really think that those who knock this sort of thing are those who have been conditioned to accept a very limited and pedestrian set of possibilities for their lives, and thus reject the limitless possibilities that imagination and exploration offer.

 

[Squid Surprise]

Now, IF the bias was correct, the image still could be "correct". But if the bias was incorrect this is most likely a totally false image. Or at least contain the same possible inaccuracies as the original data used to bias the filtering algorithm.

And my point, had you bothered to read, was that the bias IS almost certainly correct.... this "bias" comes from decades of observation and research from fairly intelligent scientists... what makes you think it's wrong?!!?

And while the software takes this "previous bias" as PART of rendering this image, it was by no means the only way. Those 8 giant telescopes still had plenty of "real" data that reinforced the image - without any bias.

Again, before you shoot down something that tons of brilliant people spent years and millions of dollars on, perhaps think that just MAYBE, they might know a bit more than you do?