Recently, I have been building a distributed web analytics system of multiple servers. This system has by its nature a repository of terabyte amounts of archived and compressed log files going back seven years. Overall, this company has 1.6 petabytes of data over all, the logs, are just a portion of that and I'm so glad I only deal with the web based logs. For those who do not know, a petabyte is a unit of information equal to one quadrillion (short scale) bytes, or 1000 terabytes; and a terabyte (TB) is 1,024 gigabytes (GB). Anyway, you get the idea. Its a lot of data.
I started working on this system about ten years ago and before that, I worked on a smaller version of it in another department. I have other responsibilities but this has been a major player in my work day. For years.
I have built this system up from a single server, to now having two separate systems/versions of about fifteen servers. Before that, I had this system running on my workstation. That seems like, decades ago now; well, it does go back to a previous millennium. I am building up a new system with new servers, and once it is ready to go, I will retire the old system. I have already started on working out the budget and details of the next system while will require all new servers with much more powerful hardware (and software, obviously). But that is for next year, or the year after depending upon budget.
But for now, I have been in the process of moving things from the old system to the new system. I have copied over the entire repository of logs and archives. Just this week, I have started to compare and delete all the archives on the old system. In fact, I'm doing that today, I may be doing this tomorrow, too. I'm telling you, its a lot of files to verify and delete. And if I lose a single log, there are indeed people who, sooner or later, will notice a gap in a historical report and want to know, where is that data and think that it is the most important thing in the world (for some reason, which is funny because this is "trending" software, not accounting software; its designed to show TRENDS people, not exactitudes! -- yes, can you tell I've had this discussion with people before?).
Which is what sparked this blog article.
Because, I noticed something as I was deleting all these files. That is, I noticed that I was deleting literally year's worth of work and effort. I have built this system from nothing, built executables that move logs around, archive them, and have had to fix it as things broke, or developers have had problems, or noticed inconsistencies which required hours, days, and in a few cases, months of working with the vendor support team.
There was one day that this old system went down, and it went down hard. The CEO, seemingly everyone at work was involved, watching, waiting for it to come back up. I don't remember what the reason was for that, but for some reason, at that time, it was being heavily used. I had worked on this system with a guy in another department, who does reporting for the ebiz side of the company, between the two of us, we have been the web analytics department for years and years. And for the first so many of those years, really not until recently, people weren't paying much attention, and no, we haven't gotten much acknowledgment for all out work. In fact, we were only told that to work on this project, was not a path to advancement. Nice huh?
Anyway, this one time, I couldn't get the system back up. I worked with the vendor support people, they couldn't get it back up. We tried to reinstall it, that didn't get it back up. I stayed late, I took off at 10PM to catch a ferry home and conclude my two hour commute home; then when I got home, I got online and started to work on it more; from time to time, my boss would call and ask for a progress report. Last time I heard form him was 2AM. I continued to work. I finally got the system up at 6AM and I told him, and went to bed. I had been up working on this almost entirely, for 24 hours straight. I was exhausted, but I had his undying thanks.
Then, not that long after that, he left the company. That is something else that happens a lot at a corporation. You work hard, prove yourself to your boss(es), and/or, you get them in your debt, then they leave and you have to start all over again; sometimes, with people who openly have no faith in you. You work hard, you try to prove you deserve your "attitude" and eventually they find out its true; but typically, not until you have beat yourself half to death in proving yourself all over again. Its really a waste of time, but it does keep you sharp. And, annoyed. Really, its not a great way to live.
Anyway, I have a lot invested in this system.
And now, I'm deleting it. That old system, like a trusty old dog, I am killing it, one delete at a time. And I noticed that, this has affected me. I was surprised to find that because in some ways, this old system has been a huge burden and problem. The database on it has gotten so messed up that even the vendor has asked that we build an entirely new system and simply dump the old one. Which, is what we're doing. The new system is running so much better. I should be glad to see the old one go.
But, I've noticed that, its almost like sawing off a piece of yourself, as I go through the process. Its, melancholy. Because, along with all this work, has gone other things in my life. I got divorced, back in 2002; my kids went through jr high, high school, started college, got jobs, its amazing really, what has taken place throughout this past decade or so.
Part of what brings all this to home, is that as I got back in time for each web server's logs, starting with the present month, and going backward, the logs get smaller. Back in 2000 we were getting many less hits. Now, we are a national power on the internet and have received awards in various ways and we are a leader in the field.
I have the history of all that in my hands. And I'm deleting it all, log by log. Yes, I have a full copy of everything on the new servers. But still, the feeling is there, I am deleting so much. I started yesterday, and I'm still going through it today. Its a lot to delete and I have to verify it so I do not delete a single file without knowing positively that there is a copy available elsewhere.
Its interesting though, how work can affect you personally, like this, outside of your work environment.
But what about my own history? I haven't deleted it, but it has passed. I have my memories, I have my family and friends. But so much is now gone. Is that bad? No, that's life, isn't it. But it pays once in a while, to stop, and reflect, on what has happened, what is happening, where you are going?
Don't you think so, too?
No comments:
Post a Comment