What is a good way to store processed CSV data to train model in Python?












1












$begingroup$


I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.



Some options appear to be:




  • HDFS

  • HDF5

  • HDFS3

  • PyArrow






python keras dataset csv serialisation







share|improve this question











$endgroup$












  • $begingroup$
    When I want to got 5 mts in distance, I would rather walk than to take a car.
    $endgroup$
    – Kiritee Gak
    17 hours ago










  • $begingroup$
    I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
    $endgroup$
    – honar.cs
    17 hours ago






  • 1




    $begingroup$
    Just leave it as CSV you don't need to do anything
    $endgroup$
    – arhwerhwe
    16 hours ago






  • 1




    $begingroup$
    Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
    $endgroup$
    – n1tk
    9 hours ago
















1












$begingroup$


I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.



Some options appear to be:




  • HDFS

  • HDF5

  • HDFS3

  • PyArrow






python keras dataset csv serialisation







share|improve this question











$endgroup$












  • $begingroup$
    When I want to got 5 mts in distance, I would rather walk than to take a car.
    $endgroup$
    – Kiritee Gak
    17 hours ago










  • $begingroup$
    I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
    $endgroup$
    – honar.cs
    17 hours ago






  • 1




    $begingroup$
    Just leave it as CSV you don't need to do anything
    $endgroup$
    – arhwerhwe
    16 hours ago






  • 1




    $begingroup$
    Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
    $endgroup$
    – n1tk
    9 hours ago














1












1








1





$begingroup$


I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.



Some options appear to be:




  • HDFS

  • HDF5

  • HDFS3

  • PyArrow






python keras dataset csv serialisation







share|improve this question











$endgroup$




I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.



Some options appear to be:




  • HDFS

  • HDF5

  • HDFS3

  • PyArrow






python keras dataset csv serialisation




python keras dataset csv serialisation






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 17 hours ago









Media

7,42262162




7,42262162










asked 17 hours ago









B SevenB Seven

21218




21218












  • $begingroup$
    When I want to got 5 mts in distance, I would rather walk than to take a car.
    $endgroup$
    – Kiritee Gak
    17 hours ago










  • $begingroup$
    I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
    $endgroup$
    – honar.cs
    17 hours ago






  • 1




    $begingroup$
    Just leave it as CSV you don't need to do anything
    $endgroup$
    – arhwerhwe
    16 hours ago






  • 1




    $begingroup$
    Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
    $endgroup$
    – n1tk
    9 hours ago


















  • $begingroup$
    When I want to got 5 mts in distance, I would rather walk than to take a car.
    $endgroup$
    – Kiritee Gak
    17 hours ago










  • $begingroup$
    I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
    $endgroup$
    – honar.cs
    17 hours ago






  • 1




    $begingroup$
    Just leave it as CSV you don't need to do anything
    $endgroup$
    – arhwerhwe
    16 hours ago






  • 1




    $begingroup$
    Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
    $endgroup$
    – n1tk
    9 hours ago
















$begingroup$
When I want to got 5 mts in distance, I would rather walk than to take a car.
$endgroup$
– Kiritee Gak
17 hours ago




$begingroup$
When I want to got 5 mts in distance, I would rather walk than to take a car.
$endgroup$
– Kiritee Gak
17 hours ago












$begingroup$
I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
$endgroup$
– honar.cs
17 hours ago




$begingroup$
I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
$endgroup$
– honar.cs
17 hours ago




1




1




$begingroup$
Just leave it as CSV you don't need to do anything
$endgroup$
– arhwerhwe
16 hours ago




$begingroup$
Just leave it as CSV you don't need to do anything
$endgroup$
– arhwerhwe
16 hours ago




1




1




$begingroup$
Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
$endgroup$
– n1tk
9 hours ago




$begingroup$
Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
$endgroup$
– n1tk
9 hours ago










3 Answers
3






active

oldest

votes


















4












$begingroup$

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.



Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.






share|improve this answer









$endgroup$









  • 1




    $begingroup$
    +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
    $endgroup$
    – Bilkokuya
    13 hours ago












  • $begingroup$
    That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
    $endgroup$
    – B Seven
    10 hours ago



















3












$begingroup$

You can find a nice benchmark for every approach in here.



enter image description here






share|improve this answer









$endgroup$





















    1












    $begingroup$

    Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.






    share|improve this answer









    $endgroup$













      Your Answer





      StackExchange.ifUsing("editor", function () {
      return StackExchange.using("mathjaxEditing", function () {
      StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
      StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
      });
      });
      }, "mathjax-editing");

      StackExchange.ready(function() {
      var channelOptions = {
      tags: "".split(" "),
      id: "557"
      };
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function() {
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled) {
      StackExchange.using("snippets", function() {
      createEditor();
      });
      }
      else {
      createEditor();
      }
      });

      function createEditor() {
      StackExchange.prepareEditor({
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: false,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: null,
      bindNavPrevention: true,
      postfix: "",
      imageUploader: {
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      },
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      });


      }
      });














      draft saved

      draft discarded


















      StackExchange.ready(
      function () {
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48008%2fwhat-is-a-good-way-to-store-processed-csv-data-to-train-model-in-python%23new-answer', 'question_page');
      }
      );

      Post as a guest















      Required, but never shown

























      3 Answers
      3






      active

      oldest

      votes








      3 Answers
      3






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      4












      $begingroup$

      With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.



      Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.






      share|improve this answer









      $endgroup$









      • 1




        $begingroup$
        +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
        $endgroup$
        – Bilkokuya
        13 hours ago












      • $begingroup$
        That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
        $endgroup$
        – B Seven
        10 hours ago
















      4












      $begingroup$

      With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.



      Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.






      share|improve this answer









      $endgroup$









      • 1




        $begingroup$
        +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
        $endgroup$
        – Bilkokuya
        13 hours ago












      • $begingroup$
        That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
        $endgroup$
        – B Seven
        10 hours ago














      4












      4








      4





      $begingroup$

      With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.



      Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.






      share|improve this answer









      $endgroup$



      With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.



      Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.







      share|improve this answer












      share|improve this answer



      share|improve this answer










      answered 17 hours ago









      Shamit VermaShamit Verma

      1,00929




      1,00929








      • 1




        $begingroup$
        +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
        $endgroup$
        – Bilkokuya
        13 hours ago












      • $begingroup$
        That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
        $endgroup$
        – B Seven
        10 hours ago














      • 1




        $begingroup$
        +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
        $endgroup$
        – Bilkokuya
        13 hours ago












      • $begingroup$
        That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
        $endgroup$
        – B Seven
        10 hours ago








      1




      1




      $begingroup$
      +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
      $endgroup$
      – Bilkokuya
      13 hours ago






      $begingroup$
      +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
      $endgroup$
      – Bilkokuya
      13 hours ago














      $begingroup$
      That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
      $endgroup$
      – B Seven
      10 hours ago




      $begingroup$
      That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
      $endgroup$
      – B Seven
      10 hours ago











      3












      $begingroup$

      You can find a nice benchmark for every approach in here.



      enter image description here






      share|improve this answer









      $endgroup$


















        3












        $begingroup$

        You can find a nice benchmark for every approach in here.



        enter image description here






        share|improve this answer









        $endgroup$
















          3












          3








          3





          $begingroup$

          You can find a nice benchmark for every approach in here.



          enter image description here






          share|improve this answer









          $endgroup$



          You can find a nice benchmark for every approach in here.



          enter image description here







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered 16 hours ago









          Francesco PegoraroFrancesco Pegoraro

          60918




          60918























              1












              $begingroup$

              Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.






              share|improve this answer









              $endgroup$


















                1












                $begingroup$

                Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.






                share|improve this answer









                $endgroup$
















                  1












                  1








                  1





                  $begingroup$

                  Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.






                  share|improve this answer









                  $endgroup$



                  Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered 17 hours ago









                  MediaMedia

                  7,42262162




                  7,42262162






























                      draft saved

                      draft discarded




















































                      Thanks for contributing an answer to Data Science Stack Exchange!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid …



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      Use MathJax to format equations. MathJax reference.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function () {
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48008%2fwhat-is-a-good-way-to-store-processed-csv-data-to-train-model-in-python%23new-answer', 'question_page');
                      }
                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      -

                      Popular posts from this blog

                      He _____ here since 1970 . Answer needed [closed]What does “since he was so high” mean?Meaning of “catch birds for”?How do I ensure “since” takes the meaning I want?“Who cares here” meaningWhat does “right round toward” mean?the time tense (had now been detected)What does the phrase “ring around the roses” mean here?Correct usage of “visited upon”Meaning of “foiled rail sabotage bid”It was the third time I had gone to Rome or It is the third time I had been to Rome

                      Image
                      Engineer refusing to file/disclose patents Extending the spectral theorem for bounded self adjoint operators to bounded normal operators Do varchar(max), nvarchar(max) and varbinary(max) columns affect select queries? Why is it that I can sometimes guess the next note? What linear sensor for a keyboard? Why don’t women say שלא עשני גויה and שפחה, according to Ashkenazim? Did arcade monitors have same pixel aspect ratio as TV sets? anything or something to eat Query about absorption line spectra Can I use my Chinese passport to enter China after I acquired another citizenship? Fuse symbol on toroidal transformer Create all possible words using a set or letters Does the Mind Blank spell prevent the target from being frightened? Can I sign legal documents with a smiley face? When quoting, must I also copy hyphens used to divide words that continue on the next line? Hot bath for aluminium engine block and heads Journal losing indexing services Is it possib
                      Read more

                      Bunad

                      Image
                      Tradisjonelle folkedrakter frå Aust-Telemark, truleg frå 1880-åra. Drakter som dette er grunnlaget for moderne bunadar. Foto: Axel Lindahl Bunad frå Eggedal i Buskerud fotografert i 1906. Bunad (frå kledebunad , der norrønt «búnaðr» har same rot som (føre)bu) er ei samnemning brukt om klesdrakter i tradisjonell nordisk stil — ofte utvikla frå folkedrakter, men likevel i ein eigen tradisjon. Ein bunad er typisk ei forseggjort og kostbar drakt, med mykje broderi, og med handsmidde smykke i gull eller sølv, særleg dei særeigne søljene, som obligatorisk prynad i tillegg. Bunad og bunadbruk er eit debattemne med mange ulike oppfatningar, til dømes om kva som er ein bunad, og om ein kan endra på utforminga av draktene eller ikkje. Den norske bunadrørsla går tilbake til folkedansrørsla og føregangsfolk som Hulda Garborg og Klara Semb. I dag blir bunad brukt som nasjonaldrakt. Festbunad er likestilt med galla-antrekk, og ofte nytta ved særskilde høve, som konfirmasjonar eller d
                      Read more

                      Færeyskur hestur Heimild | Tengill | Tilvísanir | LeiðsagnarvalRossið - síða um færeyska hrossið á færeyskuGott ár hjá færeyska hestinum

                      Image
                      FæreyjarHestar færeyskahjaltlandshestinumíslenski hesturinnBretlandskolanámumenskuWikipediafæreyskuWikipediaRossið - síða um færeyska hrossið á færeysku Færeyskur hestur Úr Wikipediu, frjálsa alfræðiritinu Jump to navigation Jump to search Rani og Grani. Grani í vetrarfeldi. Færeyski hesturinn (færeyska: Føroyska rossið ) er hestur sem hefur lifað í Færeyjum í hundruði ára. Hann er smágerður, 120-132 cm á hæð, og helst skyldur hjaltlandshestinum. Hlutverk hans var að draga vagna og plóg og bera klyfjar. Hann hefur fjórar gangtegundir eins og íslenski hesturinn (þar með talið tölt) og er með fjölda litaafbrigði. Nú til dags er hann notaður til frístunda. Í lok 19. aldar voru til um 800 hross. Mörg þeirra voru seld til Bretlands til að vinna í kolanámum og um 1960 voru aðeins nokkrir hestar eftir. Átak var gert í að varðveita hestinn og eru nú til 70-80 hross í Færeyjum. [1] Heimild |   Wikimedia Commons er með margmiðlunarefni sem te
                      Read more