Friday 11:30 a.m.–noon

Creating Bomb-Proof Data Importers

Daniel Lindsley

Audience level:
Intermediate
Category:
Best Practices & Patterns

Description

Building importers that consume other peoples' data can sometimes feel a bit like dumpster-diving. Non-compliant formats, network errors and missing data are just some of the many ways your importer is a ticking time bomb. This talk presents ways to diffuse that bomb, so when the data's broke, you can still be the hero.

Abstract

Building importers that consume other peoples' data can sometimes feel a bit like dumpster-diving. Non-compliant formats, network errors and missing data are just some of the many ways your importer is a ticking time bomb. This talk presents ways to diffuse that bomb, so when the data's broke, you can still be the hero. To help preserve your hero status, we'll be talking about: * Common pitfalls * How to structure your importer better * The importance of logging & how to manage it * Failure cases & how to address them * Processing the data We'll also talk about what tools you can use to help get the job done, how to keep it running fast/light/smooth & a few war-stories of my mistakes laid bare for some good-old public shaming.