Psychotomy

Admittedly it’s the most exercise I get most weeks, but not the point.

So, there’s a process that runs monthly. Person A does a bunch of manual data manipulation and sends the results to person B who goes through a long detailed checklist of actions that summate as “load into database using load button”

I was given the task to automate what Person B does. Right. B. Not A. And I was given the work that someone else had started before me. And that’s when I found out exactly what B does (nothing) and how much it would take to automate (it IS automated, she just clicks ‘load’ fer cryin out loud after uploading the files!!!)

Turns out C (my predecessor on the task) had started work on automating what person A does. Which is great, since apparently thats what was really wanted, despite what I was told. And C was nearly done. When I looked at the scripts, they didn’t do anything.

One of them went through ~200 lines of setup and then just printed “file processed”. There was no actual work done. Just a lot of code to get ready to do work. There was no file ever referenced. It was kindof majikal! Thats when we decided that C had somewhat misrepresented how much he had done when he left. But I worked with D and we got everything lined up, we now read the same files that A gets to begin with, we process them, and we spit out the answers. But we put it directly into the database, finishing the automation of B.

I’ve been testing this process once a month when A/B do their run, so we can make sure it all works. There’s been a few kinks. This month was supposed to be the final test b/c I finally got a meeting I’d been asking for ~6 months to happen. And got the answers I needed. And now things appeared to line up, with one exception because I didn’t have access to the unrestricted files that A had. So we got those, and then we were gonna do the Oct test. B/c the data files get overwritten daily, so I couldn’t go back and retry Sep.

I run the test Monday. And it doesn’t line up. Oh, he must have activated that new bit he was talking about (and that I found out by pure accident since i happened to get that meeting). I fix that, re-run. Same # of entries, same IDs, great. Further checking, and a bunch of column X doesn’t match. WTF? do some back digging. back some more. my datafile doesn’t have the data needed to get the answers (it has 0 instead of a useful number)

I came to that conclusion tuesday, around lunch. I have since run the process a dozen more times, I have gotten four alternative files (I now copy the entire distribution directory, and have access to most of it, so i have a few files to try, plus i got a file from A, although he does data manipulation on the original before throwing it away, so i could only work from his modified file, which didn’t match my original and had good data rather than my junk data) (all my alternative files had the same junk data)

Thursday’s solution was to quit using the datafiles and hook directly into a database that I can look the answer to X up. Which is great, except the logical piece to look up the answer for X with is X. Which is garbage. So I have to use W, which I could have used locally, except it doesn’t come up with reliable answers b/c the data format for W is non-absolute. Which means the database lookup won’t help, really.

So, basically, I’ve spent a week now trying to debug, defuse, and dodge around a data corruption error. In code. I’m trying to use code to get answers that I can’t get b/c the data is corrupted.

It’s a data corruption error! it is unreasonable to expect any amount of sane reasonable code to ‘work around’ the problem Especially since any code that gets past it will, by definition, work badly for non-garbled data. B/c the garbage is what I SHOULD use to get my answers.

And who’s to say the part I use instead isn’t going to be garbaged in 3 months instead? eesh. It’s not like it’s even that VITAL, IMO. It’s not like I’m crashing a rocket. It’s more like putting an undyed fruit loop in the box. Call it a special promomtion and get over it!

Posted in: work by attriel /