Welcome to the Amazon Redshift insert challenge (no s3 or ice buckets). So LIVE at Oracle OpenWorld 2014 #OOW14, I have been challenged by data professionals to load 1 million records into Amazon Redshift from Oracle Data Integrator WITHOUT using an S3 bucket during a live #OOW14 session: Oracle Data Integration: A Crucial Ingredient for Cloud Integration [CON7926]:
@SAsInSumit million row insert live, what could go wrong?
— Jeff Smith (@thatjeffsmith) September 12, 2014
@SAsInSumit @DataDirect_News #ODI can do many things, but this may be tough. And live? I don't know...
— Michael Rainey (@mRainey) September 12, 2014
@SAsInSumit @DataDirect_News Seeing is believing. Can't wait to actually SEE that :P
— Christophe Dupupet (@XofDup) September 12, 2014
@SAsInSumit @DataDirect_News PROVE IT! I will believe it when I see it!!!!
— Sarah CraynonZumbrum (@EPM_Queen) September 12, 2014
Beyond Redshift | Oracle and Cloud Data Integration:
Amazon Redshift is just one cloud data source, so we'll also demo and talk about connecting various cloud data sources to build an Oracle data warehouse with ODI listed below. We know this is a hot topic since my last two sessions on Salesforce data integration at Dreamforce 12 and 13 both sold out.
- Oracle Marketing Cloud (Eloqua)
- Oracle Service Cloud (RightNow)
- Salesforce.com
- Marketo
- Microsoft Dynamics
- Google Analytics / Adwords
Challenge not looking good so far ...
- ODI Job: Loading 1M records into the supplier sample table in Amazon Redshift
- Work tables are created in Redshift target for ELT
- Source data is in an Oracle 11g Database
- Data Integration platform is Oracle Data Integrator 12c
- Open source 8.4 PostgreSQL JDBC driver
- Everything is running on a local OracleVM
Since 1 million rows takes awhile, I put a filter to only load 40,288 rows which is taking 790 seconds or ~13 minutes. We're looking at close to 6 hours to load 1M at this rate.
How do I get myself out of this?
- Is it even possible to get this load within 10 minutes?
- Moscone may have flaky wifi?
- Laptop could be confiscated by TSA!
- May fall off the GG bridge during the 5K OpenWorld Bridge Run Sunday.
But I'm not worried about it since I am backed by the greatest R&D team for data connectivity in the world at Progress DataDirect; and plan to use the new DataDirect Amazon Redshift JDBC Driver.
@SAsInSumit @DataDirect_News This could be the best #ODI12c demo ever or not... Do you really think you can pull it off? #oow14
— Julien Testut (@JulienTestut) September 12, 2014
@SAsInSumit @JulienTestut @DataDirect_News Looking forward to #oow14 to see the demo succeed and collect winnings. Our tech is a safe bet :)
— Brody Messmer (@brodymessmer) September 13, 2014
If we pull this off, I will ask Julien Testut, Principal Product Manager for ODI, who is hosting the session to buy everyone in the audience a local treat (Morning Bun or Cup of Philz?).
Learn more about how DataDirect makes the impossible possible in data connectivity
Get started with a trial download today.
Or tweet to @DataDirect_News, talk to us on live chat or call 1-800-876-3101.
Sumit Sarkar
Technology researcher, thought leader and speaker working to enable enterprises to rapidly adopt new technologies that are adaptive, connected and cognitive. Sumit has been working in the data access infrastructure field for over 10 years servicing web/mobile developers, data engineers and data scientists. His primary areas of focus include cross platform app development, serverless architectures, and hybrid enterprise data management that supports open standards such as ODBC, JDBC, ADO.NET, GraphQL, OData/REST. He has presented dozens of technology sessions at conferences such as Dreamforce, Oracle OpenWorld, Strata Hadoop World, API World, Microstrategy World, MongoDB World, etc.