1. TechSpot is dedicated to computer enthusiasts and power users. Ask a question and give support. Join the community here.
    TechSpot is dedicated to computer enthusiasts and power users.
    Ask a question and give support.
    Join the community here, it only takes a minute.
    Dismiss Notice

Taking that picture of a black hole required massive amounts of data

By Cal Jeffrey · 32 replies
Apr 12, 2019
Post New Reply
  1. On Wednesday, we reported that scientists had taken the first image ever of a black hole. It was big news in the astronomy community. It was also apparently big news in the data-hoarding community as well.

    Subreddit DataHoarder was buzzing after one of the scientists working on the black hole project tweeted a picture of his colleague, Katie Bouman, posing with some of the hard drives that held the image data.

    She stood behind eight towers that appeared to hold eight drives each. Presumably, this was just one cache as 64 HDDs could only hold 5PB if they were 80TB capacity, something drive makers have not achieved yet.

    “The Black Hole photo is very impressive, but I'm more interested in where they got these 80TB drives,” said one Redditor.

    Another poster posited that they used 12TB drives and that this was only one-eighth of the data.

    "Well that makes sense as the data was captured across 8 different telescopes, so this is 1/8th of the whole setup," said ThomasTheSpider.

    The black hole image was put together using data from eight radio telescopes from around the world. Each telescope gathered massive amounts of information on its own. All totaled the scientists were dealing with more than five petabytes of data — enough to store 5,000 years worth of MP3s.

    Or as one data hoarder put it, “Black holes are cool, I guess, but imagine all the [lossless Blu-Ray files] you could store on those bad boys, in RAID 1 no less.”

    Regardless of how many drives were used, there is a logistical problem with sharing this much information. Obviously, it can’t be sent via the internet. Instead, the drives were crated and flown to Boston and Germany for processing.

    The data was then pieced together by an algorithm written by Katie Bouman. Blank spots were filled in by the software. Bouman, who is an assistant professor at the California Institute of Technology explained the process back in 2016 when she was still an MIT student (above).

    “If we want to see the black hole, we need a telescope the size of the Earth [like a disco ball],” Bouman said. In other words, mirrors covering the entire planet, or at least a hemisphere.

    However, having only eight mirrors limits what can be seen. So data was collected from each telescope every day throughout the day for an entire week.

    “As the Earth rotates, we get to see other new measurements,” she continued. “In other words, as the disco ball spins, those mirrors change locations, and we get to observe different parts of the image. The imaging algorithms we developed fill in the missing gaps of the disco ball in order to reconstruct the underlying black hole image.”

    This is why the image took up five petabytes of data. Could you imagine the data involved if it were possible to cover the Earth with mirrors to "see" the whole image?

    Permalink to story.

     
  2. FF222

    FF222 TS Addict Posts: 158   +95

    What's worse, that as explained by her in her video, this was a guided process that preferred outcomes which matched a pre-set concept of how the image should look. Which in turn means, that the black hole might not look at all like in this image, and the image was only generated this way from all the noisy data because of the built in bias in the process. So, when we'll be able to take better images of for ex. a black hole that's closer to us, it might turn out that black holes don't look at all like in this picture.
     
  3. TechGamer

    TechGamer TS Evangelist Posts: 470   +74

    This just fascinates me how technology has become so advanced quickly, remembering my childhood 20 gigabytes were considered a lot and look at the stage were at now. Time sure is flying by..
     
  4. pcnthuziast

    pcnthuziast TS Evangelist Posts: 513   +136

    Let the msm tell it and apparently Katie Bouman did it all by herself.
     
  5. seeprime

    seeprime TS Guru Posts: 355   +383

    Agreed. A team leader is not the entire team,
     
    Carljames and p51d007 like this.
  6. Jeff Re

    Jeff Re TS Addict Posts: 138   +102

    I want to know if they backed up the data before shipping it.
     
  7. Squid Surprise

    Squid Surprise TS Evangelist Posts: 2,370   +1,371

    It's not quite as abstract as you are leading us to believe... the "pre-set" concept of what a Black Hole should look like is based on tons of previously gained data... this isn't just a bunch of grade-schoolers shooting in the dark.

    It's a far more complicated process - and it's almost certainly a very accurate image. Will be nice to see the one they take of closer holes though...
     
  8. Spudnation

    Spudnation TS Rookie

    Am I the only one who thinks the image is a little underwhelming...?
     
  9. Vulcanproject

    Vulcanproject TS Evangelist Posts: 671   +897

    They didn't. The amount was so large and the cost of duplicate drives beyond the budget. What they got they flew around the world and hope it got there safely.

    The BBC did a fantastic hour long show on this entire endeavor and it aired the day the picture was revealed. It was pretty fantastic, a great deal of the hardware are was shown off.

    'How to see a black hole: The universe's greatest mystery.' Available on BBC iPlayer. Try a VPN or some such if you outside the UK.
     
  10. Lew Zealand

    Lew Zealand TS Guru Posts: 444   +332

    No no no.

    The armchair scientists here at TS totally know better than the scientists who do the actual work. It's, like, way more accurate to come up with a superficial opinion in 30 seconds than spend a decade or more of your life generating, planning, and executing experiments.
     
  11. Lew Zealand

    Lew Zealand TS Guru Posts: 444   +332

    Yeah, that was my first reaction. Here it is with a scale, which really doesn't change that impression but at least it's more than just a blob:

    https://xkcd.com/2135/
     
    Cal Jeffrey and ShagnWagn like this.
  12. VitalyT

    VitalyT Russ-Puss Posts: 4,302   +2,771

    I am aroused, that black hole does it for me. And I'm beginning to understand what managed to move Einstein in his elderly years.
     
  13. FoxxiFurr

    FoxxiFurr TS Rookie

    But they didn't take a picture of the black hole in our galaxy, they took a picture of the one in M87. This article isn't credible in any sense of the word.
     
    mbrowne5061 and JaredTheDragon like this.
  14. Lew Zealand

    Lew Zealand TS Guru Posts: 444   +332

    Eh? Reading comprehension?
    What does the black hole in our galaxy have to do with this article?
     
    Charles Olson likes this.
  15. dms96960

    dms96960 TS Guru Posts: 319   +74

    All that data for this shi**y photo? What a waste of $$$$.
     
  16. LeroN

    LeroN TS Enthusiast Posts: 84   +27

    It looks like some new kind of a scientist's fake. The presentation seems done in a kidden garden.
     
    JaredTheDragon likes this.
  17. DukeJukem

    DukeJukem TS Rookie Posts: 18   +7

    Why are we wasting money studying something that will simply kill us if it/we get close enough. it's like an ant studying a wind tunnel thats on with millions of spinning blades inside. entirely pointless and a waste of time.....wtf. cure something.
     
  18. Squid Surprise

    Squid Surprise TS Evangelist Posts: 2,370   +1,371

    Well, here's the thing.... one can never predict what advances scientific research will lead to... for instance, the invention of Velcro came from NASA scientists trying to find an adhesive material that would work in zero gravity...

    We fund all sorts of science that seems "useless" just because of things like this... for all we know, a really cool invention might spring from this...

    And it wasn't YOUR time being wasted - other than the few seconds you "wasted" reading this article... I say "seconds" because I'm fairly certain you didn't read the whole thing...
     
  19. wiyosaya

    wiyosaya TS Evangelist Posts: 3,559   +1,924

    In my early computer days, 20-megabytes was considered a lot.
     
    takemaru and Charles Olson like this.
  20. wiyosaya

    wiyosaya TS Evangelist Posts: 3,559   +1,924

    Whatever each of you used to post these messages was made possible by science. Often, science leads to the unexpected that the layman does not yet see...
     
  21. J spot

    J spot TS Maniac Posts: 217   +138

    Those were some of the most ignorant comments, surprised that such people would be on a tech website. Extreme lack of curiosity or understanding of why others would like it. And even the typical, "Why can't these astronomers cure something?" Basically people who think that these things are done fore likes on social media.

    Just the way they were able to get such a close up shot by essentially creating a huge radio telescope using the rotation of the Earth to fill in the different spots was really amazing.

    The reason it looks blurry is because it really is just a zoomed in dot inside of what would be a regular size pictere. That's how far away it is.
     
  22. Wahidsatu

    Wahidsatu TS Rookie

    Well, I guess Nokia is heading in the right direction for once.
     
  23. FF222

    FF222 TS Addict Posts: 158   +95

    Me leading you to believe? But it wasn't, but she explaining the bias process. I just made a logical conclusion based on what she said. So, IF what she says is correct, then what I'm saying is ALSO correct.

    Which if used as a bias, will ensure that we can only get results that mostly confirm the already existing data - even if reality is different. That's the actual point I made.

    That smear blob? I mean, if it would be a really crisp and detailed image, then chances would be lower that all of it was only a result of the bias. But this smear is not definitive by any means. With all the bias added to the filtering process, you could generate this smear even from totally random noise. Which is essentially what they supposedly did - except the data wasn't totally, just partially random noise, which however overwhelmed the actual data. That's the reason why the biased filtering was needed.

    Now, IF the bias was correct, the image still could be "correct". But if the bias was incorrect this is most likely a totally false image. Or at least contain the same possible inaccuracies as the original data used to bias the filtering algorithm.
     
    Last edited: Apr 14, 2019
  24. Reachable

    Reachable TS Evangelist Posts: 366   +174

    ... and there's a human need to explore. I really think that those who knock this sort of thing are those who have been conditioned to accept a very limited and pedestrian set of possibilities for their lives, and thus reject the limitless possibilities that imagination and exploration offer.
     
    wiyosaya likes this.
  25. Squid Surprise

    Squid Surprise TS Evangelist Posts: 2,370   +1,371

    And my point, had you bothered to read, was that the bias IS almost certainly correct.... this "bias" comes from decades of observation and research from fairly intelligent scientists... what makes you think it's wrong?!!?

    And while the software takes this "previous bias" as PART of rendering this image, it was by no means the only way. Those 8 giant telescopes still had plenty of "real" data that reinforced the image - without any bias.

    Again, before you shoot down something that tons of brilliant people spent years and millions of dollars on, perhaps think that just MAYBE, they might know a bit more than you do?
     
    Charles Olson and ddferrari like this.

Add your comment to this article

You need to be a member to leave a comment. Join thousands of tech enthusiasts and participate.
TechSpot Account You may also...