{"id":3936,"date":"2021-04-07T14:18:35","date_gmt":"2021-04-07T14:18:35","guid":{"rendered":"https:\/\/backupsfdc.io\/?p=3936"},"modified":"2021-04-07T14:36:46","modified_gmt":"2021-04-07T14:36:46","slug":"salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect","status":"publish","type":"post","link":"http:\/\/18.233.203.232\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/","title":{"rendered":"Salesforce Data Mastery QA: How to build for large data volumes like a Master Architect"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"3936\" class=\"elementor elementor-3936\" data-elementor-settings=\"[]\">\n\t\t\t\t\t\t\t<div class=\"elementor-section-wrap\">\n\t\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-badc298 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"badc298\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-3cf79bf2\" data-id=\"3cf79bf2\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-14979b61 elementor-widget elementor-widget-text-editor\" data-id=\"14979b61\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-text-editor elementor-clearfix\"><p><!-- wp:paragraph --><\/p>\n<h1><span style=\"font-weight: 400;\">Salesforce Data Mastery Q&amp;A: How to build for large data volumes like a Master Architect<\/span><\/h1>\n<p><em>By Susannah St-Germain, Technical Architect at Odaseva<\/em><\/p>\n<p>&nbsp;<\/p>\n<p><!-- \/wp:paragraph -->\n<iframe loading=\"lazy\" width=\"560\" height=\"630\" src=\"https:\/\/www.youtube.com\/embed\/fwrebFWtvWk\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n<!-- wp:paragraph --><\/p>\n<p>Large data volumes in Salesforce environments can grow out of control and cause problems like frustratingly slow user experiences for your employees, subpar customer experiences &#8211; even bringing down your entire Salesforce org.<\/p>\n<p>Many Salesforce admins inherit this problem when lots of data was generated by code written by multiple internal and external teams over the span of a decade or more. Now these large data volumes are suddenly overwhelming your Salesforce environment and fingers are pointing at you to fix it. It\u2019s not uncommon.<\/p>\n<p>But designing and implementing solutions is challenging when you have a large Salesforce org, and figuring out how to avoid the risks associated with large data volumes can be tricky. How do you avoid hitting Salesforce governor limits? How often should you archive? How can you reduce expensive data storage costs?\u00a0<\/p>\n<p>Odaseva <a href=\"https:\/\/www.youtube.com\/watch?v=fwrebFWtvWk\">recently hosted a webinar<\/a> to show how you can reign in the risks associated with large data volumes &#8211;\u00a0 without sacrificing business requirements. We collected and answered questions from the community about their most vexing LDV questions. The webinar, called &#8220;<em><strong>Salesforce Data Mastery Q &amp; A: How to build for large data volumes like a Master Architect<\/strong><\/em>,&#8221; featured my brilliant colleague Carl Brundage, Master Architect at Odaseva, and Anil Sistla, Platform Architect at Schneider Electric.<\/p>\n<p>Below is a summary of the questions and some key takeaways that any organization dealing with large data volumes should know:<\/p>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<h2><strong>What is a large data volume in Salesforce? What is the number when they really start to matter?<\/strong><\/h2>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<p>Large data volumes usually have the following characteristics:<\/p>\n<ul>\n<li>Large amount of records (&gt;10M records)<\/li>\n<li>Large number of fields<\/li>\n<li>Large size per record (Rich Text Field, Files) (API Size)<\/li>\n<li>Large size of data storage (&gt;50GB)<\/li>\n<\/ul>\n<p>As far as when large data volumes really start to matter &#8211; that can vary considerably. Carl&#8217;s response is what he calls the &#8220;classic architect answer.&#8221;<\/p>\n<p>&#8220;It depends,&#8221; he says. He explains that you can start to see the negative effects of large data volumes when you have fewer than 1 million records if you haven&#8217;t implemented best practices. The telltale signs are slow or even failing performance like &#8220;reports are taking a long time to load, you go to the all-list view and guess what &#8211; nothing comes back so you just wait and wait and wait.&#8221;<\/p>\n<p>So while 10 million is considered a large data volume, the negative effects can start impacting you with far fewer records if you don&#8217;t plan for it.<\/p>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<h2><strong>What are the risks associated with large data volumes in Salesforce?<\/strong><\/h2>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<p>Anil from Schneider Electric explains that large data volumes can impact system performance ranging from very slow user experiences, to directly impacting customers, even to bringing down the Salesforce org.<\/p>\n<p>&#8220;When running reporting on huge data volumes, you are going to be consuming so much resources on the platform which is going to impact your users who are trying to perform transactions on the platform,&#8221; says Anil. &#8220;Your users are starting to see the page loading taking much longer than it used to, sometimes minutes and minutes of time.&#8221;<\/p>\n<p>&#8220;The issues look simple but are actually impacting your platform on a huge scale,&#8221; he says. That\u2019s because when your employees are impacted by slowdowns, it ultimately affects your customers. &#8220;You&#8217;re directly translating that to your customer experience. Let&#8217;s say your customer care agent is on a call with a customer and if you&#8217;re taking such a long time to even open up a case, you&#8217;re directly impacting your customer who is probably not going to be willing to wait for that long.&#8221;<\/p>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<h2><strong>So how do we start to solve some of these issues? What are some best practices about mitigating these risks?<\/strong><\/h2>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<p>Carl says that &#8220;even if you\u2019ve done the right things at a code level, you can run into problems when you start moving data in and out of the system. So the question becomes what can you do about it?&#8221;<\/p>\n<p>Carl cites bypass strategies as a way to skip automations if the data coming into the system \u201calready has the other things that the automations may be doing.\u201d<\/p>\n<p>\u201cFor example&#8230; for one customer to update 100,000 cases with batch Apex it took 6-7 hours of run time and that\u2019s because of triggers, lots of logic blocks, process builders, work flows, escalation. Once a bypass strategy was implemented, it took 10 minutes.&#8221;<\/p>\n<p>So while we talk about large data volumes and working within the constraints of Salesforce, there\u2019s a lot that we can do as users to make sure that the things we&#8217;re adding on top don\u2019t impact our large data volume experience even more.<\/p>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<h2><strong>What is the best way to avoid hitting Salesforce governor limits when dealing with large data volumes?<\/strong><\/h2>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<p>There are &#8220;in-your-face&#8221; limits\u2026 and then there are &#8220;sneaky&#8221; limits.<\/p>\n<p>Susannah explains that there are API limits in Salesforce that can come as a surprise because while some Salesforce limits are soft, where you can ask for an increase, &#8220;a lot of these API limits are strictly enforced for good reason.\u00a0<\/p>\n<p>A quick example of how that might play out: if you&#8217;re not aware of the different API limits that are governed by each API\u2026 and you pick not-the-best API for the job, you can start to run into these 24 hour rolling blocks where if you exceed the number of API calls for a 24 hour rolling period you have to wait until that next period to open up and that can mean pause to your business and ultimately a pause to your customer. So that&#8217;s extremely important to be aware of.&#8221;<\/p>\n<p>But it&#8217;s not just the obvious limits that you need to be aware of, says Carl. Sneaky limits can catch you off guard, such as that you can only put 30 million files into Salesforce, or partner limits. &#8220;Looking for sneaky limits at the start will save you a lot of headaches down the road.&#8221;<\/p>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<h2><strong>Storage space in Salesforce can be pricey, what should be done to keep data volumes low? Do we move data out of Salesforce?<\/strong><\/h2>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<p>This is another question where the answer is &#8220;it depends.&#8221; There are many different options depending on the business requirements.<\/p>\n<p>Anil explains that &#8220;One aspect is storage space, the other aspect is ensuring that your users are getting the optimal performance from the Salesforce platform they&#8217;re using.&#8221; Schneider Electric deals with very, very large data volumes in the tens of millions, and Anil says that &#8220;it\u2019s a very methodical process you have to put in place, it&#8217;s not just purely from a storage standpoint but from an end to end journey that you&#8217;re trying to address for your agents or your customer.&#8221;<\/p>\n<p>&#8220;There is some data that we can take off the platform but at the same time we don\u2019t want to lose that access to the data because it\u2019s a gold mine in today&#8217;s world and it&#8217;s always important for your agents to refer back to something they have addressed in the past.&#8221;<\/p>\n<p>So Schneider Electric&#8217;s archiving goals weren&#8217;t just about taking data off the platform, but also to think about &#8220;what are we going to gain in terms of improving the agent experience, how do we still supplement the agent experience so they never feel that they lost the data and they still have access to it? The first and foremost thing that is very important is to talk to your business users, understand what is it that they need and what they&#8217;ll be using often&#8221; and then remove what can be removed based on that.<\/p>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<h2><strong>When should you archive Salesforce data?<\/strong><\/h2>\n<p><!-- \/wp:paragraph -->\n\n<!-- wp:paragraph --><\/p>\n<p>Anil explains that once you start getting into very large data volumes, it\u2019s a trigger point to start thinking about your archiving strategy. While Schneider Electric archives every day, Anil says that there&#8217;s no hard and fast rule. &#8220;It&#8217;s going back to\u2026 what is the business expectation? What&#8217;s the volume of the data growth that you have? That is what is really driving the frequency that you need to archive the data.&#8221;<\/p>\n<p>I like to sum this up as: &#8220;Data growth + business requirements = frequency.&#8221;<\/p>\n<p>&#8212;<\/p>\n<p>If you\u2019re managing Large Data Volumes in Salesforce, we can help. Architects and platform owners at Fortune 500 companies like Schneider Electric, Robert Half, and Heineken rely on Odaseva to back up, archive, and distribute huge volumes of data. If you want to level up your data management, get in touch for a <a href=\"https:\/\/www.odaseva.com\/contact-us\/\">personalized demo<\/a>.<\/p>\n<p><!-- \/wp:paragraph --><\/p><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Salesforce Data Mastery Q&amp;A: How to build for large data volumes like a Master Architect By Susannah St-Germain, Technical Architect at Odaseva &nbsp; Large data volumes in Salesforce environments can grow out of control and cause problems like frustratingly slow user experiences for your employees, subpar customer experiences &#8211; even bringing down your entire Salesforce [&hellip;]<\/p>\n","protected":false},"author":6,"featured_media":3943,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[18],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v16.1.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Salesforce Data Mastery Q&amp;A: How to build for Salesforce large data volumes<\/title>\n<meta name=\"description\" content=\"Large data volumes in Salesforce environments can cause problems. This Odaseva webinar shows how to avoid hitting Salesforce governor limits and other problems with Salesforce large data volumes\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Salesforce Data Mastery Q&amp;A: How to build for large data volumes like a Master Architect\" \/>\n<meta property=\"og:description\" content=\"Large data volumes in Salesforce environments can grow out of control and cause problems like frustratingly slow user experiences for your employees, subpar customer experiences \u2013 even bringing down your entire Salesforce org.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/\" \/>\n<meta property=\"og:site_name\" content=\"Odaseva\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-07T14:18:35+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-04-07T14:36:46+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/18.233.203.232\/wp-content\/uploads\/2021\/03\/odaseva_security_badge.png\" \/>\n\t<meta property=\"og:image:width\" content=\"936\" \/>\n\t<meta property=\"og:image:height\" content=\"930\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Salesforce Data Mastery Q&amp;A: How to build for large data volumes like a Master Architect\" \/>\n<meta name=\"twitter:description\" content=\"Large data volumes in Salesforce environments can grow out of control and cause problems like frustratingly slow user experiences for your employees, subpar customer experiences \u2013 even bringing down your entire Salesforce org.\" \/>\n<meta name=\"twitter:image\" content=\"http:\/\/18.233.203.232\/wp-content\/uploads\/2021\/03\/odaseva_security_badge.png\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\">\n\t<meta name=\"twitter:data1\" content=\"7 minutes\">\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"http:\/\/18.233.203.232\/#website\",\"url\":\"http:\/\/18.233.203.232\/\",\"name\":\"Odaseva\",\"description\":\"Odaseva - Salesforce Enterprise Data Management: Backup, Restore, Data Privacy, Data Agility\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"http:\/\/18.233.203.232\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/18.233.203.232\/wp-content\/uploads\/2021\/03\/odaseva_security_badge.png\",\"contentUrl\":\"http:\/\/18.233.203.232\/wp-content\/uploads\/2021\/03\/odaseva_security_badge.png\",\"width\":936,\"height\":930,\"caption\":\"Odaseva\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/#webpage\",\"url\":\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/\",\"name\":\"Salesforce Data Mastery Q&A: How to build for Salesforce large data volumes\",\"isPartOf\":{\"@id\":\"http:\/\/18.233.203.232\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/#primaryimage\"},\"datePublished\":\"2021-04-07T14:18:35+00:00\",\"dateModified\":\"2021-04-07T14:36:46+00:00\",\"author\":{\"@id\":\"http:\/\/18.233.203.232\/#\/schema\/person\/8183a0e13075dccb9b5c6fb1d277f7b0\"},\"description\":\"Large data volumes in Salesforce environments can cause problems. This Odaseva webinar shows how to avoid hitting Salesforce governor limits and other problems with Salesforce large data volumes\",\"breadcrumb\":{\"@id\":\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.backupsfdc.io\/\",\"url\":\"https:\/\/www.backupsfdc.io\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/\",\"url\":\"https:\/\/www.backupsfdc.io\/2021\/04\/07\/salesforce-data-mastery-qa-how-to-build-for-large-data-volumes-like-a-master-architect\/\",\"name\":\"Salesforce Data Mastery QA: How to build for large data volumes like a Master Architect\"}}]},{\"@type\":\"Person\",\"@id\":\"http:\/\/18.233.203.232\/#\/schema\/person\/8183a0e13075dccb9b5c6fb1d277f7b0\",\"name\":\"Alexandre Serrano\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"http:\/\/18.233.203.232\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/0.gravatar.com\/avatar\/3c6fefdcbbcddb040449a59359ed7ff1?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/0.gravatar.com\/avatar\/3c6fefdcbbcddb040449a59359ed7ff1?s=96&d=mm&r=g\",\"caption\":\"Alexandre Serrano\"},\"sameAs\":[\"https:\/\/www.odaseva.com\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/posts\/3936"}],"collection":[{"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/comments?post=3936"}],"version-history":[{"count":33,"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/posts\/3936\/revisions"}],"predecessor-version":[{"id":4002,"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/posts\/3936\/revisions\/4002"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/media\/3943"}],"wp:attachment":[{"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/media?parent=3936"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/categories?post=3936"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/18.233.203.232\/wp-json\/wp\/v2\/tags?post=3936"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}