How to be able to process a large JSON response?












1















We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?










share|improve this question

























  • Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

    – Jayant Das
    yesterday











  • @JayantDas I have added clarification to my question.

    – Sander de Jong
    yesterday











  • Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

    – Jayant Das
    yesterday
















1















We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?










share|improve this question

























  • Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

    – Jayant Das
    yesterday











  • @JayantDas I have added clarification to my question.

    – Sander de Jong
    yesterday











  • Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

    – Jayant Das
    yesterday














1












1








1








We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?










share|improve this question
















We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?







json rest governorlimits






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited yesterday







Sander de Jong

















asked yesterday









Sander de JongSander de Jong

1,37921538




1,37921538













  • Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

    – Jayant Das
    yesterday











  • @JayantDas I have added clarification to my question.

    – Sander de Jong
    yesterday











  • Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

    – Jayant Das
    yesterday



















  • Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

    – Jayant Das
    yesterday











  • @JayantDas I have added clarification to my question.

    – Sander de Jong
    yesterday











  • Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

    – Jayant Das
    yesterday

















Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

– Jayant Das
yesterday





Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

– Jayant Das
yesterday













@JayantDas I have added clarification to my question.

– Sander de Jong
yesterday





@JayantDas I have added clarification to my question.

– Sander de Jong
yesterday













Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

– Jayant Das
yesterday





Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

– Jayant Das
yesterday










2 Answers
2






active

oldest

votes


















4














There's just no way to cram that much data into Salesforce as-is.



I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).



As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.



Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs






share|improve this answer
























  • I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

    – Sander de Jong
    yesterday













  • I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

    – Sander de Jong
    yesterday






  • 1





    @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

    – Derek F
    yesterday



















0














In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.






share|improve this answer























    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "459"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsalesforce.stackexchange.com%2fquestions%2f255209%2fhow-to-be-able-to-process-a-large-json-response%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    4














    There's just no way to cram that much data into Salesforce as-is.



    I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).



    As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.



    Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs






    share|improve this answer
























    • I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

      – Sander de Jong
      yesterday













    • I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

      – Sander de Jong
      yesterday






    • 1





      @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

      – Derek F
      yesterday
















    4














    There's just no way to cram that much data into Salesforce as-is.



    I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).



    As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.



    Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs






    share|improve this answer
























    • I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

      – Sander de Jong
      yesterday













    • I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

      – Sander de Jong
      yesterday






    • 1





      @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

      – Derek F
      yesterday














    4












    4








    4







    There's just no way to cram that much data into Salesforce as-is.



    I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).



    As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.



    Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs






    share|improve this answer













    There's just no way to cram that much data into Salesforce as-is.



    I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).



    As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.



    Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs







    share|improve this answer












    share|improve this answer



    share|improve this answer










    answered yesterday









    Derek FDerek F

    20.7k52353




    20.7k52353













    • I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

      – Sander de Jong
      yesterday













    • I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

      – Sander de Jong
      yesterday






    • 1





      @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

      – Derek F
      yesterday



















    • I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

      – Sander de Jong
      yesterday













    • I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

      – Sander de Jong
      yesterday






    • 1





      @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

      – Derek F
      yesterday

















    I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

    – Sander de Jong
    yesterday







    I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

    – Sander de Jong
    yesterday















    I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

    – Sander de Jong
    yesterday





    I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

    – Sander de Jong
    yesterday




    1




    1





    @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

    – Derek F
    yesterday





    @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

    – Derek F
    yesterday













    0














    In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.






    share|improve this answer




























      0














      In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.






      share|improve this answer


























        0












        0








        0







        In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.






        share|improve this answer













        In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered yesterday









        sfdcfoxsfdcfox

        261k12208453




        261k12208453






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Salesforce Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsalesforce.stackexchange.com%2fquestions%2f255209%2fhow-to-be-able-to-process-a-large-json-response%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Færeyskur hestur Heimild | Tengill | Tilvísanir | LeiðsagnarvalRossið - síða um færeyska hrossið á færeyskuGott ár hjá færeyska hestinum

            He _____ here since 1970 . Answer needed [closed]What does “since he was so high” mean?Meaning of “catch birds for”?How do I ensure “since” takes the meaning I want?“Who cares here” meaningWhat does “right round toward” mean?the time tense (had now been detected)What does the phrase “ring around the roses” mean here?Correct usage of “visited upon”Meaning of “foiled rail sabotage bid”It was the third time I had gone to Rome or It is the third time I had been to Rome

            Slayer Innehåll Historia | Stil, komposition och lyrik | Bandets betydelse och framgångar | Sidoprojekt och samarbeten | Kontroverser | Medlemmar | Utmärkelser och nomineringar | Turnéer och festivaler | Diskografi | Referenser | Externa länkar | Navigeringsmenywww.slayer.net”Metal Massacre vol. 1””Metal Massacre vol. 3””Metal Massacre Volume III””Show No Mercy””Haunting the Chapel””Live Undead””Hell Awaits””Reign in Blood””Reign in Blood””Gold & Platinum – Reign in Blood””Golden Gods Awards Winners”originalet”Kerrang! Hall Of Fame””Slayer Looks Back On 37-Year Career In New Video Series: Part Two””South of Heaven””Gold & Platinum – South of Heaven””Seasons in the Abyss””Gold & Platinum - Seasons in the Abyss””Divine Intervention””Divine Intervention - Release group by Slayer””Gold & Platinum - Divine Intervention””Live Intrusion””Undisputed Attitude””Abolish Government/Superficial Love””Release “Slatanic Slaughter: A Tribute to Slayer” by Various Artists””Diabolus in Musica””Soundtrack to the Apocalypse””God Hates Us All””Systematic - Relationships””War at the Warfield””Gold & Platinum - War at the Warfield””Soundtrack to the Apocalypse””Gold & Platinum - Still Reigning””Metallica, Slayer, Iron Mauden Among Winners At Metal Hammer Awards””Eternal Pyre””Eternal Pyre - Slayer release group””Eternal Pyre””Metal Storm Awards 2006””Kerrang! Hall Of Fame””Slayer Wins 'Best Metal' Grammy Award””Slayer Guitarist Jeff Hanneman Dies””Bullet-For My Valentine booed at Metal Hammer Golden Gods Awards””Unholy Aliance””The End Of Slayer?””Slayer: We Could Thrash Out Two More Albums If We're Fast Enough...””'The Unholy Alliance: Chapter III' UK Dates Added”originalet”Megadeth And Slayer To Co-Headline 'Canadian Carnage' Trek”originalet”World Painted Blood””Release “World Painted Blood” by Slayer””Metallica Heading To Cinemas””Slayer, Megadeth To Join Forces For 'European Carnage' Tour - Dec. 18, 2010”originalet”Slayer's Hanneman Contracts Acute Infection; Band To Bring In Guest Guitarist””Cannibal Corpse's Pat O'Brien Will Step In As Slayer's Guest Guitarist”originalet”Slayer’s Jeff Hanneman Dead at 49””Dave Lombardo Says He Made Only $67,000 In 2011 While Touring With Slayer””Slayer: We Do Not Agree With Dave Lombardo's Substance Or Timeline Of Events””Slayer Welcomes Drummer Paul Bostaph Back To The Fold””Slayer Hope to Unveil Never-Before-Heard Jeff Hanneman Material on Next Album””Slayer Debut New Song 'Implode' During Surprise Golden Gods Appearance””Release group Repentless by Slayer””Repentless - Slayer - Credits””Slayer””Metal Storm Awards 2015””Slayer - to release comic book "Repentless #1"””Slayer To Release 'Repentless' 6.66" Vinyl Box Set””BREAKING NEWS: Slayer Announce Farewell Tour””Slayer Recruit Lamb of God, Anthrax, Behemoth + Testament for Final Tour””Slayer lägger ner efter 37 år””Slayer Announces Second North American Leg Of 'Final' Tour””Final World Tour””Slayer Announces Final European Tour With Lamb of God, Anthrax And Obituary””Slayer To Tour Europe With Lamb of God, Anthrax And Obituary””Slayer To Play 'Last French Show Ever' At Next Year's Hellfst””Slayer's Final World Tour Will Extend Into 2019””Death Angel's Rob Cavestany On Slayer's 'Farewell' Tour: 'Some Of Us Could See This Coming'””Testament Has No Plans To Retire Anytime Soon, Says Chuck Billy””Anthrax's Scott Ian On Slayer's 'Farewell' Tour Plans: 'I Was Surprised And I Wasn't Surprised'””Slayer””Slayer's Morbid Schlock””Review/Rock; For Slayer, the Mania Is the Message””Slayer - Biography””Slayer - Reign In Blood”originalet”Dave Lombardo””An exclusive oral history of Slayer”originalet”Exclusive! Interview With Slayer Guitarist Jeff Hanneman”originalet”Thinking Out Loud: Slayer's Kerry King on hair metal, Satan and being polite””Slayer Lyrics””Slayer - Biography””Most influential artists for extreme metal music””Slayer - Reign in Blood””Slayer guitarist Jeff Hanneman dies aged 49””Slatanic Slaughter: A Tribute to Slayer””Gateway to Hell: A Tribute to Slayer””Covered In Blood””Slayer: The Origins of Thrash in San Francisco, CA.””Why They Rule - #6 Slayer”originalet”Guitar World's 100 Greatest Heavy Metal Guitarists Of All Time”originalet”The fans have spoken: Slayer comes out on top in readers' polls”originalet”Tribute to Jeff Hanneman (1964-2013)””Lamb Of God Frontman: We Sound Like A Slayer Rip-Off””BEHEMOTH Frontman Pays Tribute To SLAYER's JEFF HANNEMAN””Slayer, Hatebreed Doing Double Duty On This Year's Ozzfest””System of a Down””Lacuna Coil’s Andrea Ferro Talks Influences, Skateboarding, Band Origins + More””Slayer - Reign in Blood””Into The Lungs of Hell””Slayer rules - en utställning om fans””Slayer and Their Fans Slashed Through a No-Holds-Barred Night at Gas Monkey””Home””Slayer””Gold & Platinum - The Big 4 Live from Sofia, Bulgaria””Exclusive! Interview With Slayer Guitarist Kerry King””2008-02-23: Wiltern, Los Angeles, CA, USA””Slayer's Kerry King To Perform With Megadeth Tonight! - Oct. 21, 2010”originalet”Dave Lombardo - Biography”Slayer Case DismissedArkiveradUltimate Classic Rock: Slayer guitarist Jeff Hanneman dead at 49.”Slayer: "We could never do any thing like Some Kind Of Monster..."””Cannibal Corpse'S Pat O'Brien Will Step In As Slayer'S Guest Guitarist | The Official Slayer Site”originalet”Slayer Wins 'Best Metal' Grammy Award””Slayer Guitarist Jeff Hanneman Dies””Kerrang! Awards 2006 Blog: Kerrang! Hall Of Fame””Kerrang! Awards 2013: Kerrang! Legend”originalet”Metallica, Slayer, Iron Maien Among Winners At Metal Hammer Awards””Metal Hammer Golden Gods Awards””Bullet For My Valentine Booed At Metal Hammer Golden Gods Awards””Metal Storm Awards 2006””Metal Storm Awards 2015””Slayer's Concert History””Slayer - Relationships””Slayer - Releases”Slayers officiella webbplatsSlayer på MusicBrainzOfficiell webbplatsSlayerSlayerr1373445760000 0001 1540 47353068615-5086262726cb13906545x(data)6033143kn20030215029