Search Content
find
CATALOG
Data Infrastructure
Data Tools
Desktops, Laptops and OS
Enterprise Applications
IT Infrastructure
IT Management
Networking and Communications
Security
Servers and Server OS
Software and Web Development
Storage
Dashboard
News
My ITmodelbook
Messages
Events (0)
CONNECTIONS
more
Connect
with other members or
invite
your contacts to the community.
+ invite connections
GROUPS
more
Join
an existing group to participate in the group discussions or
create
a new group of your own to create discussions around topics of interest to you and your work.
+ create group
MY RESOURCES
more
Find
info or
add
new info to your ITmodelbook. You can find white papers, technology reports, business analysis, webinars, presentations and more. You can also share your own authored content and resources you like by adding this info.
+ add resource
Online Users=0
Resources you may like
Debatching Bulk Data a Free 33 Page Chapter
Share within ITmb
Share This Resource
Share with: Connections
or
Share with: Groups
Debatching data is the process of turning one huge pile of data into many small piles of data. Why is it better to shovel one ton of data using two thousand, one pound shovels instead of one big load from a huge power shovel? After all, large commercial databases and the attendant bulk loader or SQL Loader programs are designed to do just that: insert huge loads of data in a single shot. The bulk load approach works under certain tightly constrained circumstances. They are as follows:
The "bulk" data comes to you already matching the table structure of the destination system. Of course, this may mean that it was debatched before it gets to your system.
The destination system can accept some, potentially significant, error rate when individual rows fail to load.
There are no updates or deletes, just inserts.
Your destination system can handle bulk loads. Certain systems (for example, some legacy medical systems or other proprietary systems) cannot handle bulk operations.
As the vast majority of data transfer situations will not meet these criteria, we must consider various options. First, one must consider which side of the database event horizon one should perform these tasks. One could, for example, simply dump an entire large file into a staging table on SQL Server, and then debatch using SQL to move the data to the "permanent" tables. There are, of course, multiple tools for debatching large bulk data loads including BizTalk Server and SQL Server Integration Services (SSIS). One can use such tools to break up large batches of data, manipulate it as needed, and send it on to its next reincarnation (for example, into an API, a relational database, or a text file). In this chapter, Packt Enterprise will take a look at options for processing large data sets.
Packt Enterprise books can be summed up through the tagline "Professional Expertise Distilled". They take in-the-trenches knowledge from experienced software professionals, and distil it into a single, easy to follow manuscript.
Level
Select Level
Subject Area
Select Subject Area
Keywords
Packt Enterprise, Debatching Bulk Data â Free 33 Page Chapter,
Offered by
Packt Enterprise
URL
Files
The resource is available from the link above.
Bookmark to
My ITmodelbook
add
Group ITmodelbooks
Latest reports from top IT companies:
SAP
HP
Janrain
HubSpot
PrepLogic
Motorola
BNP Media
Informatica
Microsoft
Jobvite
© ITmodelbook 2012-2025.
sitemap
about
privacy
terms
help
do not sell my personal information